Chaos testing 429s, retries, and tenant isolation before they become lost revenue

429 is a control signal, not a generic error

RFC 6585 defines 429 Too Many Requests for rate limiting and allows a Retry-After header describing how long to wait. A client that immediately retries converts the provider's protection signal into more load.^[1]

A safe policy classifies failures. A validation error is permanent. A 429 may be retriable after the provider's delay. A timeout has an unknown side-effect state. A 503 may indicate overload. A partially streamed response may be impossible to replay invisibly.

Timeouts should fit inside the user-operation budget. Retries should be bounded, use exponential backoff with jitter, and happen at one chosen layer. AWS documents these patterns because synchronized retries can magnify a small failure into a full outage.^[3][4]

Retries multiply across layers

Google SRE gives a concrete warning: if database, backend, frontend, and JavaScript layers each make four total attempts, one user action can create 64 database attempts. Retrying at several layers multiplies attempts instead of adding them.^[2]

The same chapter models a backend capable of 10,000 QPS receiving 10,100 QPS. One hundred rejected requests retry, raising the next second to 10,200; the retry load grows while useful first-attempt work shrinks. A retry budget contains that amplification.

The chart reproduces the documented multiplicative structure, not an observed CapacityLab customer incident.

One noisy tenant should not consume every worker

Shared connection pools, worker queues, model-provider quotas, and retry budgets create cross-tenant coupling. A single client that overloads one method can occupy concurrency needed by unrelated customers even when their own traffic is normal.

Bulkheads partition scarce resources. Per-tenant concurrency, queue capacity, token budgets, and fair scheduling keep one overload local. Global admission control still protects the service when total safe capacity is reached.

Idempotency prevents a retry from becoming a duplicate purchase or job. Stripe stores the first result for an idempotency key, while AWS describes client request identifiers as a foundation for safe retryable APIs.^[6][5]

• Per-operation concurrency caps before the shared dependency pool.
• Per-tenant queue and retry budgets with fair scheduling.
• Idempotency keys persisted through timeout and worker restart.
• Circuit breaker state that does not synchronize every caller into one probe storm.
• Dead-letter and expiry behavior for work that can no longer meet the user promise.

Translate the fault into revenue exposure

A technical error budget becomes actionable when it is connected to product operations. If 120 contribution-producing operations arrive each minute at an $18 contribution margin, a complete fifteen-minute outage exposes $32,400. This is not booked loss: retries, delayed conversion, refunds, and customer recovery change the result.

The point of the model is prioritization. A 429 on image transformation may degrade presentation; a 429 on payment confirmation can hold inventory and revenue. Fault tests should follow business criticality, not infrastructure fashion.

Grafana k6 documents xk6-disruptor for fault injection, and Google SRE explicitly recommends testing overload behavior and graceful degradation. Run the test with steady traffic before the fault, during the fault, and through recovery.^[7][2]