Resilience, Retries, Backpressure, and Overload Control - Complete Notes

A language-neutral guide to containing partial failures, budgeting time and retry load, controlling queues and concurrency, surviving overload, recovering dependencies safely, and explaining every trade-off in production and system-design interviews.

00. The resilience mental model

Resilience is the ability to keep useful, correct work flowing when parts of a system are slow, unavailable, overloaded, or uncertain.

Resilience is not the absence of failure. Networks delay messages, machines stop, processes pause, callers disconnect, and traffic exceeds forecasts. The design goal is to keep one fault from consuming unrelated resources or turning into a positive feedback loop. A small visible error is often safer than waiting until every request, thread, connection, and dependency fails together.

01. Failure classes and uncertainty

A retry or fallback is safe only after the failure has been classified by duration, scope, and whether the operation might already have taken effect.

Partial failure changes the answer

In one process, a function either returns, fails, or is still running. Across a network, the caller cannot directly know whether the remote process never received the request, is still working, or committed and lost the response. A timeout is a statement about the caller's patience, not proof that the remote operation failed. This uncertainty is why writes need idempotency, reconciliation, or an outcome-query API.

Name the failure scope

• Request-local: one malformed or unusually expensive request.
• Instance-local: one process has a leak, pause, deadlock, or bad connection pool.
• Partition or shard: one key range is hot or unavailable.
• Zone or region: power, network, or control-plane failure affects a location.
• Dependency-wide: every caller shares one overloaded database or identity service.
• Correlated: a rollout, certificate expiry, or shared library fails everywhere together.

Redundancy helps only against faults outside the replicas' shared failure domain. Ten replicas do not protect against one bad configuration pushed to all ten, and they may send ten times the retry traffic to a shared dependency.

02. Timeouts, deadlines, and cancellation

A timeout bounds one wait; a deadline bounds the entire operation; cancellation stops work whose result is no longer useful.

Timeout types

Budget time down the call graph

Client deadline: 900 ms

Gateway validation and queue:       80 ms
Order service local work:            70 ms
Inventory call, all attempts:       350 ms
Pricing call, all attempts:         220 ms
Response serialization and network: 80 ms
Reserve for variance and cleanup:   100 ms
                                     ------
Total:                               900 ms

Pass the remaining deadline, not the original duration, to downstream calls. If 300 ms has already elapsed, the remaining work has 600 ms. Reject before starting an expensive subtask when the remaining time cannot cover its minimum useful execution. Use a monotonic clock for elapsed-time calculations so wall-clock corrections do not create negative or extended timeouts.

HANDLE(request, absolute_deadline):
  remaining = absolute_deadline - monotonic_now()

  if remaining < minimum_useful_budget:
    return DEADLINE_EXCEEDED

  child_deadline = min(absolute_deadline, monotonic_now() + inventory_budget)
  result = inventory.reserve(request, child_deadline)

  if caller_cancelled():
    cancel(result.in_flight_work)
    return CANCELLED

  return result

Selecting a timeout

1. Start from the user or business deadline, not a library default.
2. Measure the dependency's latency distribution under expected load and failure conditions.
3. Choose an acceptable false-timeout rate, such as a small fraction above measured p99.9.
4. Add realistic network, handshake, cold-connection, and cross-region variance.
5. Keep enough time for backoff, a second attempt when eligible, cleanup, and response delivery.
6. Revalidate after releases and capacity changes. A timeout is an operational policy.

Cancellation is cooperative

A caller stopping its wait does not magically interrupt database work, CPU loops, queued jobs, or child RPCs. Propagate cancellation through each boundary and check it before expensive work and inside long loops. Close response bodies, return connections, release permits, and cancel child calls in a guaranteed cleanup path. Some irreversible work may need to finish and reconcile even after the caller leaves.

03. Retry eligibility and idempotency

A retry is a new load-bearing request. Make it only when the operation is safe to repeat, the failure may be temporary, and enough deadline and retry budget remain.

Idempotent effect, not merely an HTTP verb

An operation is idempotent when repeating the same logical command produces the same intended effect as applying it once. A read is often naturally idempotent. A create, debit, notification, or inventory decrement is not unless the protocol gives every logical operation stable identity and stores the outcome atomically with the effect.

PROCESS(command, idempotency_key, authenticated_principal):
  identity = (principal, operation_type, idempotency_key)

  begin transaction
    prior = outcomes.find(identity)
    if prior exists:
      if hash(command) != prior.command_hash:
        return KEY_REUSED_WITH_DIFFERENT_COMMAND
      return prior.response

    validate business invariants
    effect = apply_once(command)
    outcomes.insert(identity, hash(command), effect.response)
  commit transaction

  return effect.response

Scope the key to the authenticated actor and operation. Store a command digest so an attacker or bug cannot reuse one key for different parameters. Define retention long enough to cover delayed retries, secure the stored response, and treat concurrent duplicates atomically. Idempotency does not mean the server must return byte-identical timestamps or trace IDs, but the business effect must not repeat.

Ambiguous write timeline

Time    Client                     Inventory service              Database
t0      send reserve K-42  ------>
t1                                  begin transaction ---------->
t2                                                               decrement stock
t3                                                               store outcome K-42
t4                                                               commit
t5                                  send success -X- connection breaks
t6      timeout: outcome UNKNOWN
t7      retry reserve K-42 ------->
t8                                  find stored outcome
t9      <------------------------- return original success

Without K-42, the retry can decrement stock twice.

04. Bounded retries, backoff, jitter, and budgets

Exponential backoff

capped_exponential(attempt) = min(cap, base * 2^attempt)

full_jitter(attempt) = random(0, capped_exponential(attempt))

equal_jitter(attempt) =
  capped_exponential(attempt) / 2
  + random(0, capped_exponential(attempt) / 2)

decorrelated_jitter(previous) =
  min(cap, random(base, previous * 3))

Exponential backoff reduces attempt frequency while a dependency recovers. A cap prevents impractically long sleeps. Jitter breaks synchronization among clients that failed at the same time. Full jitter spreads attempts most widely; equal jitter preserves a minimum pause; decorrelated jitter avoids a rigid schedule. Do not use the request ID as a permanently stable random seed that causes the same clients to collide on every incident.

A complete retry policy has six bounds

1. Eligible operation and error classes.
2. Maximum total attempts, including the first attempt.
3. Maximum elapsed time under the original deadline.
4. Backoff schedule with jitter and a cap.
5. A process, client, or dependency retry budget.
6. Cancellation and server pushback behavior.

CALL_WITH_RETRY(operation, deadline):
  previous_delay = base_delay

  for attempt from 1 to max_attempts:
    if cancelled() or remaining(deadline) < minimum_attempt_time:
      return DEADLINE_EXCEEDED

    result = call(operation, timeout = attempt_slice(deadline))

    if result.success:
      return result

    if not retryable(result) or not operation.is_idempotent:
      return result

    if attempt == max_attempts or not retry_budget.try_take(1):
      return result

    delay = server_pushback(result)
            or decorrelated_jitter(previous_delay)
    delay = min(delay, remaining(deadline) - minimum_attempt_time)
    wait_until(delay, cancellation)
    previous_delay = delay

  return LAST_FAILURE

Retry budgets

An attempt limit bounds one logical request. A retry budget bounds the aggregate behavior of many requests. One practical policy allows retries to be only a small percentage of original traffic, such as 10 retries per 100 first attempts over a rolling window, with a small burst allowance. When failures rise, the budget empties and clients stop adding load. Track original attempts and retries separately so success after retry does not hide dependency instability.

Retry amplification

Browser:  3 total attempts
Gateway:  3 total attempts for each browser attempt
Service:  3 total attempts for each gateway attempt

Worst-case dependency attempts = 3 * 3 * 3 = 27

If "3 retries" means 4 total attempts:
worst case = 4 * 4 * 4 = 64

Prefer retries at one layer that understands the operation and end-to-end deadline. Disable or coordinate hidden library, proxy, service-mesh, queue, and application retries. Report total attempts in tracing so multiplicative behavior is visible.

05. Hedged requests and tail latency

Hedging sends a second copy before the first fails and uses the first acceptable response.

t0       send attempt A to replica 1
t0+80ms no response, tail threshold crossed
t0+80ms send attempt B to a different failure domain
t0+105ms B succeeds
t0+106ms cancel A and return B

Both attempts share one deadline and one logical operation identity.

Hedging can reduce tail latency when a small fraction of replicas are independently slow. It adds load even when the first attempt would eventually succeed, so delay the hedge near a measured tail percentile, select another failure domain, cap attempts, share a retry budget, and cancel losers. Do not hedge overloaded dependencies, large uploads, expensive queries, non-idempotent effects, or failures that are correlated across replicas. A hedge to the same saturated shard is duplicate harm, not redundancy.

06. Circuit breakers

A circuit breaker temporarily stops calls that are unlikely to succeed, protecting both caller and dependency while allowing controlled probes for recovery.

                 failure threshold crossed
       +-------------------------------------------+
       |                                           v
   CLOSED --------------------------------------> OPEN
     ^  |                                           |
     |  | normal calls                              | cool-down expires
     |  | record eligible outcomes                  v
     |  +-------------------------------------- HALF_OPEN
     |                                               |
     +------------- probe successes ----------------+
                         |
                         +-- probe failure --> OPEN

• Closed: calls flow and eligible outcomes update a rolling window.
• Open: calls fail fast, degrade, or use a safe fallback.
• Half-open: a small, bounded number of probes test recovery.

Breaker policy

Count timeouts, connection failures, and dependency overload responses when they indicate dependency health. Usually exclude caller cancellation, validation errors, permission failures, and business rejections. Use a minimum sample size plus a rolling time or count window. A breaker per destination, operation, and sometimes tenant prevents one slow endpoint from blocking every operation. Add jitter to reopen times so all clients do not probe together.

07. Bulkheads and dependency isolation

A bulkhead partitions resources so one workload cannot consume them all. Separate concurrency pools, connection pools, queues, memory budgets, CPU allocations, rate quotas, or deployments by workload and dependency. Isolation trades utilization efficiency for predictable failure scope.

Do not create a thread pool for every call without modeling memory and scheduling cost. The principle is bounded ownership. In event-loop systems, separate semaphores, queues, and connection pools can provide isolation without a thread-per-request model.

08. Fallbacks, brownouts, and graceful degradation

A fallback must be cheaper, safe, bounded, and meaningfully better than an honest failure.

A brownout intentionally disables optional expensive features while core functions remain available. Trigger it from measured pressure with hysteresis so it does not switch rapidly. Keep the path exercised through canaries or drills, make degraded responses explicit to users and telemetry, and define which product and compliance promises still apply.

09. Bounded queues and queueing mechanics

A queue moves waiting from one place to another. It absorbs a finite burst, but cannot create sustained capacity.

arrival rate lambda = 1,200 requests/s
service rate mu     = 1,000 requests/s
net growth          =   200 requests/s

10,000 empty queue slots fill in:
10,000 / 200 = 50 seconds

After that, work must block, degrade, spill durably, or be rejected.

Little's Law gives L = lambda * W for a stable system: average in-system work equals throughput times average time in the system. At 1,000 completed requests/s and 200 ms average response time, about 200 requests are in flight on average. If latency rises to 2 seconds at the same throughput, in-flight work rises to about 2,000, consuming more memory, sockets, and permits.

Queue policy

• Bound both item count and bytes because request sizes vary.
• Give queued items deadlines and discard work that can no longer finish usefully.
• Separate priority classes and reserve capacity for control and recovery traffic.
• Use FIFO when age fairness matters, but prevent head-of-line blocking by expensive work.
• Use per-tenant limits so one producer cannot occupy the whole queue.
• Persist only when delayed completion is part of the contract, not to hide overload.
• Expose queue depth, oldest age, enqueue rate, wait time, rejection, and cancellation.

10. Backpressure across boundaries

Backpressure is effective only when the producer can observe the signal and respond. A pull model asks for a bounded number of items. A credit or window model grants permission for bytes or messages. A bounded blocking queue pauses an in-process producer. An HTTP server can return 429 or 503 with pushback. A broker consumer can reduce fetch demand or pause partitions. If the external producer cannot slow, the system must buffer within a bound, sample, aggregate, spill durably, or drop according to policy.

subscriber grants credit = 32 items
publisher sends at most 32 items
subscriber processes 16 items
subscriber grants 16 more credits

outstanding demand = total granted - total delivered

Publisher invariant:
delivered items must never exceed granted demand.

Transport flow control is not application backpressure. TCP can stop bytes entering one receiver, but the sender may still create logical work, buffer elsewhere, or hold expensive state. Carry demand and cancellation at the application level and bound buffers at every hop.

11. Admission control and load shedding

Admission control decides whether work may enter. Load shedding removes work that would otherwise push the system beyond a safe operating region.

Protect every meaningful choke point

Shed by cost, value, and deadline

1. Drop invalid, abusive, duplicate, expired, or already-cancelled work first.
2. Pause batch refreshes, prefetching, analytics, and optional fan-out.
3. Disable expensive nonessential fields or ranking stages.
4. Preserve authenticated control traffic, health reporting, and recovery operations.
5. Apply fair per-tenant or per-user limits to remaining work.
6. Reject new low-priority work early and cheaply when capacity is exhausted.

Return a precise outcome. HTTP 429 usually communicates a caller or quota limit. HTTP 503 usually communicates temporary service unavailability or overload. A Retry-After field can communicate minimum delay, but clients must still add jitter, respect their deadline, and avoid retrying unsafe operations. Do not claim a guaranteed recovery time unless the server can support it.

Hysteresis and pressure signals

Enter overload mode at a high threshold and leave only after pressure stays below a lower threshold for a recovery period. This prevents rapid mode changes. Prefer direct saturation signals such as in-flight work, queue age, event-loop lag, memory, and dependency permits over CPU alone. CPU can look low when threads are blocked on a saturated database, and high CPU can still be healthy when useful throughput remains stable.

ADMIT(request):
  if invalid(request) or request.deadline_expired:
    reject_without_queueing()

  class = authenticated_priority(request)

  if tenant_inflight(request.tenant) >= tenant_limit(class):
    return OVERLOADED_WITH_PUSHBACK

  if system_pressure > enter_brownout_threshold:
    disable_optional_work(request)

  if dependency_permits.try_acquire(class) == false:
    return safe_fallback_or_overload()

  return ACCEPTED_WITH_PERMIT

12. Concurrency limits and sustainable capacity

Rate limits control starts per unit time. Concurrency limits control simultaneous resource holders. Slow failures make concurrency the critical bound.

If a dependency normally handles 1,000 requests/s at 50 ms, average concurrency is about 50. If it slows to 500 ms while arrivals remain 1,000/s, concurrency tends toward 500. Without a limit, the caller may consume ten times the sockets, memory, threads, and downstream work before rate changes at all.

Adaptive limit intuition

An adaptive controller can increase the limit slowly while latency remains near the measured no-queue baseline, then reduce it quickly when queueing delay rises. This resembles additive increase and multiplicative decrease. It must enforce a tested minimum and maximum, ignore noisy tiny samples, separate operations with different costs, and react more slowly than transient measurement noise. A control loop that every instance runs identically can synchronize, so add jitter and canary changes.

every control_interval with enough samples:
  queueing_ratio = observed_latency / no_queue_baseline

  if errors_due_to_overload or queueing_ratio > high_threshold:
    limit = max(min_limit, floor(limit * 0.8))
  else if queueing_ratio < low_threshold:
    limit = min(max_limit, limit + 1)

  publish limit, sample count, baseline, and decision reason

Capacity must survive failure

Suppose an inventory tier has 12 instances. Load tests show each instance preserves its latency objective up to 400 requests/s, but operational policy uses only 70% of that limit. Normal safe fleet capacity is 12 * 400 * 0.70 = 3,360 requests/s. Losing one zone containing four instances leaves eight, or 2,240 requests/s at the same safety level. A 3,000 requests/s peak fits normally but does not fit after zone loss. The design needs more baseline instances, lower admitted load, a safe degraded mode, or independently proven rapid failover capacity.

13. Cascading failures and positive feedback

A cascade begins when one failure increases pressure elsewhere, causing additional failures that feed back into the original pressure.

dependency slows
      |
      v
in-flight calls and queues grow
      |
      v
memory, connections, threads, and CPU saturate
      |
      v
timeouts and health-check failures rise
      |
      +----> clients retry ----+
      |                        |
      +----> instances removed |
                               v
                     less effective capacity
                               |
                               +----> dependency slows further

Failure timeline: retries turn slowdown into outage

Why apparently helpful mechanisms worsen outages

14. Health checks, readiness, and draining

Health checks are control-plane inputs with data-plane consequences. They must be cheap, bounded, independently schedulable, and hard to fail because of ordinary request overload.

Liveness should usually test local progress, not every remote dependency. If a shared database fails, restarting every caller adds connection storms and removes otherwise useful degraded capacity. Readiness can consider local overload, but simultaneous removal of many instances can overload those left. Use failure and success thresholds, minimum ready capacity, draining, and rate-limited membership changes. Keep probe traffic outside the saturated user queue when possible, but do not let it falsely report health while the process does no useful work.

Graceful drain

1. Stop advertising readiness for new ordinary work.
2. Keep liveness true while the process drains.
3. Stop accepting new long-lived streams and tell clients to reconnect elsewhere.
4. Wait for in-flight work within a bounded termination grace period.
5. Cancel or persist unfinished work according to its contract.
6. Close listeners and resource pools only after admission stops.

15. End-to-end resilient inventory design

Assume an online retailer receives 3,000 checkout reservation requests/s at peak. Each reservation must not decrement stock twice, 99.9% should finish within 900 ms when healthy, and optional product recommendations may disappear during overload.

Authenticated client
       |
       v
Edge quota and abuse control
       |
       v
Checkout API
  | deadline propagation
  | per-tenant admission
  | optional recommendation brownout
  v
Inventory client bulkhead
  | bounded waiters and concurrency
  | one retry owner with budget
  | circuit breaker
  v
Zone-local load balancer
       |
       +----- Inventory instance A -----+
       +----- Inventory instance B -----+--> sharded inventory database
       +----- Inventory instance C -----+    plus idempotency outcomes

Metrics, traces, change events, and audit logs cross every boundary.

Normal reservation flow

1. The client creates one unpredictable idempotency key per logical checkout action.
2. The gateway authenticates the caller, enforces payload and tenant quotas, and sets a deadline.
3. Checkout validates locally and removes optional work if brownout is active.
4. The inventory bulkhead acquires a tenant and dependency permit or rejects without queueing.
5. The client sends the remaining deadline and stable operation identity to inventory.
6. Inventory atomically checks the prior outcome, verifies stock, changes stock, and stores outcome.
7. Cancellation stops optional and read work, but a committing reservation finishes and reconciles.
8. Every permit, timer, connection, and response body is released in cleanup.

Capacity reasoning

The 396-slot estimate uses p99 rather than average latency as a conservative first pass, but load testing must validate the entire distribution and resource mix. With a zone lost, 400 slots leave almost no headroom. The incident policy disables hedges, spends only a tiny retry budget, sheds low-priority reservations, and ramps failover capacity slowly. Merely routing all 3,300 attempts/s to the remaining fleet would operate at the edge of the measured limit with no safety margin.

Overload flow

t0       inventory latency rises above baseline
t0+5s    adaptive concurrency reduces new in-flight work
t0+8s    queue reaches age threshold; new low-priority work gets 503 + pushback
t0+10s   retry budget drains; only original idempotent attempts continue
t0+12s   breaker opens for the affected shard after sufficient samples
t0+15s   checkout removes recommendation and coupon fan-out
t0+20s   operators pause batch reservations and confirm database pressure falls
t0+40s   half-open probes begin with jitter and limited concurrency
t0+60s   useful throughput is stable; admitted traffic ramps in small steps

Users see some explicit rejections, but core capacity remains alive.

Go client example: deadline, permit, and one retry owner

The principle comes first: one call graph owns retry policy, every attempt shares one deadline, and a semaphore limits dependency concurrency. The following approved-ecosystem sketch uses Go's standard cancellation model. Production code also needs typed errors, response cleanup, observability, jitter, and an aggregate retry budget.

func Reserve(ctx context.Context, cmd Command) (Result, error) {
    ctx, cancel := context.WithTimeout(ctx, 350*time.Millisecond)
    defer cancel()

    select {
    case inventoryPermits <- struct{}{}:
        defer func() { <-inventoryPermits }()
    case <-ctx.Done():
        return Result{}, ctx.Err()
    default:
        return Result{}, ErrOverloaded
    }

    first, err := callInventory(ctx, cmd)
    if err == nil || !isRetryable(err) || !cmd.HasIdempotencyKey() {
        return first, err
    }

    delay := time.Duration(rand.Int64N(int64(40 * time.Millisecond)))
    timer := time.NewTimer(delay)
    defer timer.Stop()

    select {
    case <-timer.C:
        return callInventory(ctx, cmd)
    case <-ctx.Done():
        return Result{}, ctx.Err()
    }
}

TypeScript server example: early overload rejection

A server should reject before allocating expensive state. This example shows the mechanism without defining resilience through a framework. An actual service must use atomic permit accounting, trusted tenant identity, payload limits, structured logging, and disconnect cancellation.

async function handleReserve(request: Request): Promise<Response> {
  const tenant = authenticateTenant(request);
  const permit = limiter.tryAcquire(tenant);

  if (!permit) {
    return new Response("temporarily overloaded", {
      status: 503,
      headers: { "Retry-After": "1" }
    });
  }

  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), 300);

  try {
    const result = await reserveAtomically(request, controller.signal);
    return Response.json(result);
  } finally {
    clearTimeout(timer);
    permit.release();
  }
}

16. Security, privacy, abuse, and trust boundaries

Resilience controls are security controls because attackers can intentionally consume the same scarce resources as accidental overload. Authenticate before granting expensive capacity when possible, but protect authentication itself because password hashing, token introspection, key lookup, and policy checks can be expensive dependencies.

• Limit headers, body bytes, decompressed size, field counts, nesting, and streaming duration.
• Apply per-principal, per-tenant, per-IP, and global controls with deliberate proxy trust rules.
• Do not trust a client-supplied priority header without authorization.
• Bind idempotency keys to authenticated identity and command digest; use unguessable keys.
• Rate-limit outcome lookup to prevent key probing and cross-tenant inference.
• Keep overload responses free of internal host, capacity, topology, or tenant details.
• Fail closed for authorization and integrity decisions; use bounded stale only when policy permits.
• Reserve capacity for revocation, incident control, audit, and administrative recovery paths.
• Redact payloads and identifiers in retry, queue, trace, and idempotency telemetry.
• Protect emergency breaker, brownout, and limit overrides with least privilege and audit logs.

Fairness under overload

Global first-come, first-served allows one large tenant or botnet to crowd out everyone. Hierarchical limits can reserve a global safety bound, allocate tenant shares, and permit controlled borrowing of unused capacity. Decide whether fairness means equal users, paid quotas, business criticality, or cost-weighted work. Normalize for operation cost so one expensive search is not counted like one cheap health read. Never let commercial priority bypass integrity or emergency controls.

17. Metrics, tracing, dashboards, and alerts

Observe offered load, admitted load, useful completions, and rejected work separately. A service can look busy while useful throughput collapses.

Trace one logical request across attempts

Preserve one logical operation ID and create a child span for each attempt. Record attempt number, original or retry or hedge, remaining deadline, selected destination, queue wait, permit wait, server pushback, breaker state, cancellation, and final business outcome. Avoid high-cardinality raw idempotency keys and tenant identifiers in metric labels. Use sampled, access-controlled traces for detailed investigation.

Actionable alerts

Alert on user harm and sustained dangerous pressure, not every breaker transition or rejected request. Overload controls are expected to reject some work. A useful alert includes the affected service objective, scope, recent changes, dashboard, and recovery playbook.

18. Recovery playbook for an overloaded dependency

Recovery is a controlled reduction of offered work followed by a measured restoration of useful capacity. Fix the deepest failing layer first and avoid changing many feedback loops at once.

Phase 1: detect and stabilize

1. Declare an incident owner and record a timeline of changes and actions.
2. Confirm user impact, affected operations, regions, shards, tenants, and first failing layer.
3. Compare original traffic, retries, queue age, in-flight work, ready capacity, and completions.
4. Freeze deployments, rebalancing, large maintenance, and nonessential configuration changes.
5. Stop or sharply reduce retries and hedges at the owning layer.
6. Pause batch, prefetch, backfill, analytics, and other low-priority producers.
7. Enter tested brownout mode and reject excess work early at upstream choke points.
8. If instances crash-loop, reduce admitted traffic enough for a meaningful fleet fraction to live.

Phase 2: repair the limiting resource

1. Identify whether CPU, memory, locks, storage, connections, a hot key, or a recent change is limiting.
2. Revert the triggering release or configuration when evidence supports it.
3. Add capacity only if the dependency and its downstream resources can accept it safely.
4. Quarantine a bad shard, query, tenant, or request shape when the scope is narrow and verified.
5. Keep readiness and liveness changes deliberate; do not restart every slow caller automatically.
6. Verify useful throughput rises and queue age falls, not only CPU or error rate.

Phase 3: restore gradually

1. Let instances start, establish pools, elect leaders, and warm critical caches under low traffic.
2. Use a tiny, jittered half-open probe population to verify correctness and latency.
3. Ramp admitted traffic in steps while watching saturation and useful completions.
4. Restore interactive features before batch and background work.
5. Restore retry budgets last and confirm attempts do not multiply across layers.
6. Reconcile ambiguous reservations and duplicate outcomes before declaring data recovery complete.

Phase 4: learn and harden

1. Preserve evidence, causal timeline, trigger, feedback loops, and mitigation effects.
2. Write a blameless post-incident review with owners and due dates.
3. Add the incident workload to regression, overload, and chaos tests.
4. Adjust limits from measured capacity, including zone loss and cold recovery.
5. Remove temporary overrides, confirm audit records, and update the playbook.

19. Production and deployment practices

• Give every network call an explicit deadline derived from an end-to-end budget.
• Inventory library, proxy, mesh, broker, and application retries before choosing one owner.
• Keep retry, hedge, breaker, queue, and concurrency policies specific to operation cost.
• Store resilience policy as reviewed configuration with safe bounds, ownership, and change history.
• Canary policy changes and compare original traffic, useful throughput, and tail latency.
• Roll out clients gradually because a client retry change can overload a healthy server fleet.
• Keep zone-local traffic by default and reserve measured cross-zone failover capacity.
• Warm caches, pools, JIT-compiled paths, and connections before advertising full readiness.
• Separate batch from serving work and make pausing every producer operationally simple.
• Document overload response codes, pushback, idempotency, and deadline semantics in contracts.
• Keep control, health, and recovery traffic small and protected from ordinary tenant exhaustion.
• Rehearse rollback and brownout paths so rare code does not decay unnoticed.

Safe resilience-policy rollout

1. Measure the current policy and name the problem the change should solve.
2. Validate bounds statically, such as maximum attempts and minimum deadline reserve.
3. Shadow-compute the new decision without enforcing it when practical.
4. Canary on a small client and server population across representative workloads.
5. Observe at least one peak and one dependency fault scenario.
6. Expand gradually with an automatic stop on objective burn or amplification.
7. Retain a fast, audited rollback that does not depend on the failing service.

Performance, scalability, and cost trade-offs

Measure cost per successful logical operation, not cost per attempt. A retry that eventually succeeds can still triple database work. Include reserved failure capacity, rejected traffic, warm standby, observability, data reconciliation, and operator time when comparing designs.

20. Testing strategy and chaos experiments

Test until and beyond the capacity knee. A resilience mechanism is incomplete until its degraded mode and recovery behavior are verified under realistic concurrency.

Unit and property tests

• Use a controllable monotonic clock for deadline, backoff, jitter, and breaker transitions.
• Verify every retryable error and every explicitly non-retryable error.
• Property-test that attempt count and elapsed time never exceed configured bounds.
• Verify server pushback overrides local backoff only within the original deadline.
• Test breaker minimum sample, rolling window, open, half-open, and jittered recovery.
• Prove semaphore permits, response bodies, timers, and queue slots release on every exit.
• Test admission priority, tenant fairness, brownout hysteresis, and fail-safe defaults.
• Race duplicate idempotency keys and prove exactly one business effect is stored.
• Reject reuse of an idempotency key with different authenticated actor or command digest.

Integration and protocol tests

• Delay DNS, connect, TLS, response headers, body reads, and application processing separately.
• Break the connection before send, during send, after commit, and during response.
• Verify remaining deadline and cancellation propagate through every service hop.
• Return HTTP 429, 503, Retry-After, malformed pushback, and permanent errors.
• Exercise real load-balancer readiness removal, drain, and reconnection behavior.
• Test duplicate requests arriving simultaneously at different instances and zones.
• Fill each pool and queue independently to prove bulkhead isolation.
• Verify breaker and limit configuration remains compatible across rolling versions.

Load, stress, spike, and soak tests

• Ramp gradually to find the latency knee, maximum useful throughput, and exhausted resource.
• Jump instantly to peak to reveal cold-cache, connection, and admission behavior.
• Hold offered load above capacity and verify useful throughput remains stable while excess sheds.
• Run mixed cheap and expensive operations to test cost-aware fairness.
• Run one noisy tenant beside normal tenants and measure isolation.
• Soak below peak to expose leaks, slow queue growth, token drift, and breaker oscillation.
• Lose a zone at peak and verify the remaining capacity model rather than only failover routing.
• Measure recovery time after load falls, including queue drain, cache warm-up, and retry decay.

Chaos experiment catalog

Start in a representative test environment, then use a tiny production scope with explicit owner, hypothesis, blast radius, observability, abort threshold, and rollback. Never inject an ambiguous write fault without a verified reconciliation path. A chaos test that only confirms alerts is not enough; verify user impact, containment, and recovery.

Recovery testing

• Start from a crash-loop with only 10% of instances ready and determine safe admitted traffic.
• Verify batch producers stay paused while interactive service recovers.
• Test half-open probe coordination when thousands of clients share one dependency.
• Warm pools and caches under stepwise traffic and measure pressure at every step.
• Reconcile all ambiguous commands and prove no duplicate business effects remain.
• Restore normal policy, remove overrides, and confirm alert and audit closure.

21. Troubleshooting decision table

22. Common mistakes and safer corrections

23. Hands-on exercises and design scenarios

Exercise 1: Retry policy review

A mobile client, gateway, order service, and database driver each make three total attempts. Draw the worst-case attempt tree. Expected reasoning: one action can cause 81 database attempts if four layers each make three attempts. Choose one semantic owner, disable hidden retries elsewhere, keep a shared deadline, and use an aggregate retry budget.

Exercise 2: Deadline allocator

Allocate a 700 ms search deadline across gateway work, query parsing, three parallel shards, ranking, and response transfer. Expected reasoning: parallel shards share a wall-clock slice rather than adding their budgets; reserve final response time; pass remaining time; cancel stragglers; and return partial results only if the contract marks them clearly and safely.

Exercise 3: Ambiguous payment outcome

Inject a connection loss immediately after the payment database commits. Expected reasoning: the client cannot infer failure from timeout. Use one authenticated idempotency key, atomic outcome storage, an outcome-query API, duplicate-race tests, retention, and reconciliation. Do not issue an automatic refund until the original outcome is known.

Exercise 4: Queue capacity

A worker completes 800 jobs/s and receives 1,000 jobs/s for 90 seconds. Jobs average 20 KB and must start within 10 seconds. Expected reasoning: backlog grows by 18,000 jobs and about 360 MB, but the deadline allows only about 8,000 jobs at 800/s. A larger queue violates usefulness; shed, pause the producer, or add verified capacity before the oldest age reaches 10 seconds.

Exercise 5: Concurrency estimation

An API handles 5,000 requests/s at 80 ms average, then a dependency slows to 600 ms. Expected reasoning: average in-flight work grows from about 400 to 3,000 if arrivals remain constant. Bound dependency concurrency, reject before queues become useless, and measure tail latency and bytes, not only the average.

Exercise 6: Health-check cascade

During a database slowdown, readiness fails on half the API instances and liveness restarts them. Expected reasoning: deep probes coupled caller health to dependency health; capacity removal and cold restarts intensify overload. Keep liveness local, make readiness deliberate, shed traffic, protect minimum ready capacity, and drain instead of restart where possible.

Exercise 7: Multi-tenant overload design

One tenant sends expensive export requests while all tenants use interactive reads. Expected reasoning: authenticate, assign cost weights, isolate export and interactive pools, enforce tenant and global concurrency, persist exports only if asynchronous completion is contractual, reserve control capacity, and test noisy-neighbor isolation.

Exercise 8: Recovery game day

Start with 90% of inventory instances crash-looping under a retry storm. Expected reasoning: freeze changes, stop retries and batch work, admit a very small critical fraction, stabilize and warm the fleet, repair the trigger, ramp in steps, restore optional work, then retries, and reconcile ambiguous writes. Record abort criteria and recovery time.

24. Interview questions and model answers

1. What is the difference between a timeout and a deadline?

A timeout is a maximum duration for one wait or attempt. A deadline is the absolute end of the whole logical operation. A strong design propagates remaining deadline through queues, retries, and child calls, so each hop cannot restart the user's entire budget.

Follow-up: What clock should measure elapsed time?

Use a monotonic clock locally. Across RPC boundaries, protocols can transmit remaining duration or compensate for elapsed time to avoid relying on perfectly synchronized wall clocks.

2. Which failures should be retried?

Retry only when the error can plausibly be transient, the operation is idempotent or deduplicated, enough deadline remains, pushback permits it, and both attempt and aggregate retry budgets allow it. Do not retry invalid input, permission failures, exhausted deadlines, or preconditions that need a state change.

3. Why is a timed-out write ambiguous?

The caller knows only that it did not receive a timely response. The server may never have received the request, may still be processing it, or may have committed and lost the response. Resolve this with stable operation identity, atomic outcome storage, and an outcome lookup or reconciliation process.

4. Explain exponential backoff and jitter.

Exponential backoff increases delay between attempts, reducing pressure during a persistent fault. A cap keeps delay bounded. Jitter randomizes the schedule so clients that failed together do not retry together. It does not make an unsafe or overloaded retry safe, so eligibility and budgets still come first.

5. What is a retry budget?

It is an aggregate limit on retries across many requests, often expressed as a small fraction of original attempts over a window. Per-request attempt limits cannot stop millions of clients from each making one retry. A budget drains as failures rise and prevents retries from consuming all remaining capacity.

6. When is request hedging appropriate?

For idempotent, relatively cheap operations whose tail delay is rare and independent across replicas. Send a delayed second attempt near a measured tail percentile, choose another failure domain, share the original deadline and budget, and cancel losers. Avoid it during shared overload, expensive work, or ambiguous writes.

7. How does a circuit breaker work?

Closed passes calls and measures eligible failures. Open fails fast for a cool-down. Half-open admits a bounded probe set; sufficient success closes it and failure reopens it. Scope it by dependency and operation, require a minimum sample, add jitter, and do not treat it as a substitute for proactive concurrency control.

8. What is the bulkhead pattern?

It partitions resources so one failure cannot consume everything, such as separate connection and concurrency pools for checkout and analytics. The cost is stranded capacity and more operations. The correct boundary follows workload priority, dependency, tenant, and failure domain.

9. Why does an unbounded queue reduce resilience?

It hides sustained overload as growing wait. Requests can expire while queued, memory grows, and recovery takes longer because obsolete work remains. Bound item count, bytes, age, and priority; then reject, degrade, or slow producers when the bound is reached.

10. Backpressure versus load shedding?

Backpressure asks upstream to reduce production and works only when the signal propagates and is obeyed. Load shedding rejects or drops selected work when demand still exceeds safe capacity. A system often needs both: slow controllable producers and shed uncontrollable excess.

11. Rate limit versus concurrency limit?

A rate limit bounds starts per time and is useful for quotas and bursts. A concurrency limit bounds simultaneous resource holders and responds directly to slow operations. At a constant arrival rate, a tenfold latency increase can cause tenfold concurrency, so resilient services usually need both.

12. How do you choose a concurrency limit?

Load test representative work to find the useful-throughput and latency knee, then reserve headroom for bursts and failures. Split by operation cost and dependency. A fixed limit is a safe start; an adaptive controller may adjust within tested bounds using queueing delay and errors.

13. How can health checks create a cascade?

If normal overload delays deep probes, instances become unready or restart. Their traffic moves to fewer instances, which then fail too. Use cheap local liveness, separate readiness, thresholds, minimum ready capacity, and early shedding. Do not restart every caller because a shared dependency is down.

14. Why can adding replicas worsen an outage?

Failover shifts traffic to remaining replicas that may lack headroom. New replicas may be cold and create cache-fill or connection storms. More caller replicas can also amplify load on a fixed database. Model capacity after failure, warm slowly, and limit downstream concurrency globally.

15. What makes a fallback safe?

It preserves required correctness, authorization, and privacy; costs less than the primary path; has its own capacity bound and deadline; clearly communicates degraded semantics; and is regularly tested. Returning another tenant's cache, assuming a payment succeeded, or flooding the database is not a fallback.

16. How would you detect a retry storm?

Separate logical requests, first attempts, retries, and hedges. Look for attempts rising faster than original traffic, retry-budget exhaustion, lower first-attempt success, growing dependency load, and repeated attempts in traces. Mitigate by stopping retries at non-owning layers and reducing offered load.

17. How do you recover a crash-looping overloaded service?

Stop retries and nonessential producers, sharply reduce admitted traffic, repair or revert the trigger, let a meaningful fraction of capacity start and warm, then ramp traffic in small measured steps. Restore optional work and retry budgets last. Verify useful throughput and data reconciliation, not only process health.

18. Design resilience for a checkout inventory call.

Clarify latency, oversell, and availability promises. Use an end-to-end deadline, authenticated idempotency key, atomic stored outcome, zone-aware routing, per-tenant and dependency concurrency, a short bounded queue, one budgeted retry owner, a per-shard breaker, and explicit overload responses. Degrade optional features, preserve audit evidence, monitor attempts and useful throughput, test ambiguous commits and zone loss, and document staged recovery.

Follow-up: Would you serve stale inventory?

Stale inventory can be a browsing hint, but final reservation must validate authoritative stock. The answer depends on the business cost of oversell and whether a pending order state is supported.

25. Revision cheat sheet

• Resilience contains faults, preserves useful work, and enables controlled recovery.
• Classify transient, permanent, partial, slow, ambiguous, and corrupt failures.
• A timeout bounds one wait; a deadline bounds the whole operation.
• Cancellation must propagate and application work must cooperate.
• A timeout does not prove a write failed; use atomic idempotency or outcome lookup.
• Retry only eligible operations and errors within attempt, time, and aggregate budgets.
• Use capped exponential backoff with jitter and respect server pushback.
• Retries at multiple layers multiply, so choose one semantic owner.
• Hedging targets rare independent tail delay and must cancel losing attempts.
• Circuit breaker states are closed, open, and half-open; probes must be bounded.
• Bulkheads isolate pools, queues, tenants, workloads, and failure domains.
• A fallback must be safe, cheaper, bounded, explicit, and exercised.
• A queue absorbs a finite burst but cannot add sustained service capacity.
• Bound queue count, bytes, age, priority, and tenant share.
• Backpressure slows producers; load shedding rejects unavoidable excess.
• Rate limits control starts; concurrency limits control simultaneous resource use.
• Size capacity for zone loss and cold recovery, not only normal average load.
• Protect useful throughput beyond the queueing knee by rejecting early.
• Cheap local liveness and deliberate readiness avoid capacity-removal cascades.
• Retries, replicas, caches, health checks, and autoscaling can amplify outages.
• Observe original load, attempts, useful completions, queues, limits, and rejection reasons.
• Recover by cutting load, repairing the deepest fault, warming, and ramping gradually.
• Chaos tests need a hypothesis, narrow blast radius, abort rule, and recovery proof.

26. Primary official references

• IETF RFC 9110: HTTP Semantics - safe and idempotent methods, 503 Service Unavailable, and Retry-After semantics.
• IETF RFC 6585: Additional HTTP Status Codes - 429 Too Many Requests and optional retry guidance.
• gRPC official deadline guide - deadline selection, server cancellation, and deadline propagation.
• gRPC official cancellation guide - cooperative cancellation and propagation through a call graph.
• gRPC official retry guide - retry policy, throttling, backoff, and server pushback.
• gRPC official request hedging guide - delayed parallel attempts, throttling, deadlines, and cancellation.
• Amazon Builders' Library: Timeouts, retries, and backoff with jitter - timeout selection, retry amplification, token budgets, backoff, and jitter.
• Amazon Builders' Library: Using load shedding to avoid overload - admission, overload rejection, fairness, and operating beyond capacity.
• Google SRE Book: Addressing Cascading Failures - resource exhaustion, retry amplification, queue control, degradation, and recovery.
• Google SRE Book: Handling Overload - client-side admission, request cost, and graceful overload behavior.
• Reactive Streams official specification - asynchronous demand, mandatory non-blocking backpressure, and bounded buffers.
• Kubernetes official probe documentation - startup, liveness, and readiness behavior plus cascading-failure cautions.
• Go standard library: context - request-scoped deadlines, cancellation signals, and propagation.
• Java SE Flow API - standard Publisher, Subscriber, Subscription, and demand interfaces.