On 18 April 2023, I gave my first large conference talk at ArgoCon, a CNCF-hosted event alongside KubeCon Europe in Amsterdam.
I had five minutes and one story: How to Avoid a Kubernetes Doom Loop.
The incident behind the talk involved a Kubernetes cluster of 165 nodes carrying roughly 16,000 Argo Workflows. A mistake in automation kept generating work. The resulting load made the system slower, which encouraged more retries and created still more work.
That is a doom loop: a failure response that amplifies the conditions that caused the failure.
The talk attracted more than 280 registrations. I suspect that was because the pattern was familiar. The technologies vary, but many distributed-system incidents share the same uncomfortable shape.
The first failure is rarely the whole incident
Automation is normally introduced to make a task repeatable. A controller observes a difference, a worker performs an action and the system moves towards the desired state.
That model assumes several things:
- the desired state is valid;
- producers cannot create work without limit;
- workers have enough capacity to make progress;
- failure is distinguishable from slowness;
- retries allow recovery rather than add pressure;
- an operator can interrupt the process safely.
When those assumptions fail together, a controller can continue doing exactly what it was designed to do while making recovery less likely.
The first error might be a configuration mistake. The incident grows because the surrounding system has no effective boundary.
A queue is a control surface, not a storage cupboard
A queue separates the rate at which work is requested from the rate at which it can be completed. That makes it one of the most useful protections in an automation system—but only when its behaviour is visible and bounded.
Teams need to know:
- how quickly new work enters the queue;
- how quickly workers complete it;
- the age of the oldest item;
- how retries are represented;
- which tenant or producer is creating demand;
- what happens when the queue reaches a safe limit.
Queue depth alone is not enough. A stable queue of old, failing work can be worse than a larger queue that is draining predictably.
Backpressure must also reach the producer. If the consumer slows down while the producer continues at full speed, the queue has postponed the incident rather than prevented it.
Concurrency is a reliability setting
Parallelism is attractive because it shortens a healthy run. It also determines how quickly a bad instruction can be multiplied.
Set concurrency at several levels:
- a global maximum that protects the cluster;
- a per-workload or per-tenant maximum that limits noisy neighbours;
- a worker maximum based on real downstream capacity;
- a separate allowance for recovery and operational work.
The last control is easily missed. If automation consumes every schedulable resource, the tools needed to diagnose or repair the system may compete with the incident itself.
Separating critical delivery workers from the workloads they create—through dedicated node pools, priorities or other isolation—preserves a route back into the system.
Retries need a budget
A retry is useful when a failure is likely to be temporary and the next attempt has a reasonable chance of succeeding.
It is harmful when the cause is persistent, the delay is too short or every failed item retries at the same moment.
A deliberate retry policy includes:
- a maximum number of attempts;
- exponential backoff;
- jitter to avoid synchronised retry storms;
- classification of retryable and terminal failures;
- a time budget after which human judgement is required;
- idempotency, so another attempt does not duplicate side effects.
“Retry forever” is not resilience. It is an unbounded producer with a friendly name.
Design a brake before needing one
Every powerful automation path needs a safe way to pause.
That might be a feature flag, a suspended workflow, a scaled-down controller or a blocked producer. The mechanism matters less than four operational properties:
- people know it exists;
- the permission to use it is available during an incident;
- stopping new work does not corrupt work already in progress;
- the recovery sequence has been tested.
A kill switch invented during an outage is just another untested change.
Recovery also needs an order. Stop the producer. Stabilise the control plane and workers. Measure the backlog. Remove or replay work deliberately. Restore capacity in stages. Keep watching the signals that showed the loop beginning.
Observe the feedback loop
Traditional resource dashboards show the damage after amplification has begun. The more valuable signals describe the loop itself:
- work creation rate versus completion rate;
- queue age and depth;
- API-server latency and rejected requests;
- controller reconciliation and error rates;
- workflow creation, retry and termination rates;
- pending pods and scheduler pressure;
- the availability of the operational path.
Alerting on a single threshold can still miss the pattern. A rising creation rate, falling completion rate and increasing retry rate together tell a much clearer story.
Test the failure response
Happy-path load testing proves throughput. Reliability testing asks what the system does when a dependency slows down, permissions change, a producer emits invalid work or a worker repeatedly fails.
The important question is not simply, “Does the workflow fail?”
It is, “What does the automation do next, and is that response bounded?”
Run those tests below the scale at which the answer becomes expensive. Verify that limits engage, backpressure is visible, alerts reach the right people and the brake works.
What I carried away from ArgoCon
Five minutes forced the lesson into a simple shape:
Automation should make the safe action repeatable without making the unsafe action infinite.
The incident was a technical failure, but the prevention is socio-technical. Limits need owners. Alerts need responders. A stop mechanism needs authority. Recovery needs a shared sequence rather than a collection of clever commands.
Speaking at ArgoCon also changed how I talk about reliability. A polished reference architecture is less useful than an honest failure story with controls somebody can apply on Monday.
You can view the original ArgoCon session, watch the five-minute recording or revisit the original event-day LinkedIn post.