Observability, SRE, Capacity Planning, and Incident Response - Complete Notes

A language-neutral guide to producing trustworthy evidence, defining reliability promises, predicting capacity, detecting fast error-budget burn, and leading distributed-system incidents from first alert through verified recovery and lasting improvement.

00. Mental model and precise terminology

Observability turns a running system's externally visible evidence into answers about its internal state. SRE turns those answers into an engineering process for delivering a measured level of reliability.

01. Signals: logs, metrics, traces, profiles, and events

No single signal is sufficient. Each compresses reality differently and answers a different class of question.

Structured logs

Emit a stable record with timestamp, severity, event name, service, version, environment, operation, outcome, duration, trace and span identifiers, and safe domain identifiers. Prefer fields such as outcome="timeout" over parsing prose. Keep a human-readable message, but do not make it the only queryable data. Log once at the boundary that can add meaning. If five layers log the same exception, volume rises while evidence does not.

{
  "timestamp": "2026-07-18T09:42:17.481Z",
  "severity": "WARN",
  "event": "inventory_reservation_finished",
  "service": "order-api",
  "service_version": "2026.07.18.2",
  "operation": "POST /orders",
  "outcome": "dependency_timeout",
  "duration_ms": 812,
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "tenant_hash": "t_91c24",
  "region": "ap-south",
  "retry_attempt": 1
}

Focused Go example: one meaningful boundary record

This Go standard-library example records one domain outcome at the boundary that understands it. time.Since uses Go's monotonic component when the start value contains one, so a wall clock correction does not distort elapsed duration. A production tracing library should extract and inject W3C context; do not write a partial trace-header parser in each handler.

func reserveInventory(
    ctx context.Context,
    logger *slog.Logger,
    store InventoryStore,
    orderID string,
) error {
    started := time.Now()
    err := store.Reserve(ctx, orderID)

    outcome := "success"
    if err != nil {
        outcome = "failure"
    }

    logger.InfoContext(ctx, "inventory reservation finished",
        "event", "inventory_reservation_finished",
        "outcome", outcome,
        "duration_ms", time.Since(started).Milliseconds(),
        "operation", "reserve_inventory",
    )
    return err
}

The raw order ID is deliberately absent because telemetry policy may classify it as sensitive and high-cardinality. The logger can be enriched with a safe trace ID by trusted middleware. A metric instrument should count outcome and observe duration using the same bounded operation name, while the trace records the individual call.

Metrics and distributions

• Counter: cumulative events, such as requests or failures. Calculate a rate over time.
• Gauge: a value that rises and falls, such as queue depth or active requests.
• Histogram: observations aggregated into buckets, count, and sum. It supports fleet aggregation and percentile estimates.
• Summary: client-side quantiles in some systems. Precomputed quantiles generally cannot be meaningfully averaged across instances.

Use base units, explicit names, and semantic conventions. Initialize expected zero-valued series so absence does not look like zero activity. Never average averages without their counts. A 2 ms average from 10 requests and a 200 ms average from 10,000 requests do not have equal weight.

Traces, spans, and links

A trace represents a distributed operation. A span represents one timed operation with a parent, start and end times, attributes, events, status, and resource identity. Parent-child edges model a call path. A span link models a causal relationship that is not a strict parent, such as a batch consumer processing records produced by several traces. Record meaningful logical operations, not every function. A database span should identify the database system and operation safely, but should not contain secrets or raw unbounded statements.

Profiles

Continuous low-overhead profiles reveal CPU hot paths, allocations, lock contention, garbage collection, and off-CPU waits that request telemetry may not explain. Correlate profiles with the exact service version and time range. Compare a degraded window against a normal baseline. A CPU flame graph can prove where samples occurred, but it does not prove why a user request waited if the thread was blocked off-CPU. Use runtime events, lock profiles, and traces together.

02. Correlation IDs and context propagation

Evidence becomes much more valuable when a request keeps a safe causal identity across process, queue, thread, and asynchronous boundaries.

A correlation ID groups records chosen by the application. A trace ID identifies a distributed trace; a span ID identifies one operation inside it. They can coexist. Do not replace an existing valid trace context with a new ID at every hop. The W3C Trace Context recommendation defines the traceparent and tracestate HTTP headers. Validate incoming values, create a new trace when invalid, and treat caller-provided sampling flags as advice rather than authority.

Mobile client
  traceparent: trace=T1, parent=S1
       |
       v
API gateway span S2
       |
       +---- HTTP ----> Order span S3 ---- SQL ----> Database span S4
       |
       +---- publish message M, attach trace context and message_id
                              |
                              v
                     Notification consumer span S5
                     link S5 to producer context when processing is delayed or batched

Propagate the remaining deadline and cancellation separately from trace context. Propagate only allowlisted baggage. Baggage travels downstream and may cross an unexpected trust boundary, has no built-in integrity guarantee, and does not automatically become a span attribute. Never place credentials, raw personal data, payment details, or authorization decisions in baggage. A tenant hint from an untrusted caller must not decide access control.

Asynchronous edge cases

• Capture context when work is submitted, not when a pooled worker eventually starts it.
• Detach background work from an expired request when it has an independent lifecycle.
• For retries, preserve one logical trace but create a span for each physical attempt.
• For fan-out, create sibling spans and retain cancellation. Do not reuse one mutable span concurrently.
• For queues, record producer, broker wait, consumer, message identity, attempt, and age.
• For batch consumption, use links when one consumer operation represents multiple producer contexts.

03. Telemetry pipeline, sampling, cardinality, and retention

The telemetry path is another distributed system. It needs capacity, security, backpressure, failure visibility, and explicit loss behavior.

Application and infrastructure
  | SDK buffers and batches within strict memory bounds
  v
Local or node collector
  | receive - enrich - redact - batch - retry
  v
Regional gateway collectors
  | tail sample - route - protect tenant quotas
  +--------------+----------------+----------------+
  v              v                v                v
Metric store   Log store       Trace store      Profile store
  |              |                |                |
  +--------------+-------- query, dashboards, alerts, investigations

Prefer asynchronous bounded export so an unavailable backend cannot block business requests. Track records accepted, exported, retried, refused, and dropped at every hop. Disk buffering can bridge a short outage but creates persistence, encryption, cleanup, and full-disk risks. Decide whether to drop oldest, newest, or low-priority telemetry when buffers fill. Business service should usually continue while clearly reporting observability loss.

Head and tail sampling

Never sample security audit records solely because traffic is high. Keep a known sampling probability and carry it into analysis. If every error is retained but only 1% of successes is retained, the stored error fraction is not the real error rate. Metrics computed before sampling remain the preferred SLI source.

Cardinality is multiplicative

Metric cardinality is the number of unique label combinations. With 8 routes, 5 status classes, 4 regions, 3 versions, and 20 tenants, one metric can create 8 * 5 * 4 * 3 * 20 = 9,600 time series. Add an unbounded user ID or request ID and the state can grow with traffic. High cardinality consumes SDK memory, network, storage, query CPU, and dashboard time.

• Metrics: use bounded dimensions such as normalized route, status class, region, and version.
• Traces and logs: keep request-level identity where policy allows and retention is controlled.
• Never label metrics with raw URL paths, timestamps, stack traces, message IDs, or user IDs.
• Set SDK and backend cardinality limits, then alert on overflow rather than silently trusting partial series.

Retention and telemetry tiers

Define ownership, purpose, geographic location, retention, access, encryption, deletion, and cost for each dataset. Redact as close to the source as possible and again at controlled pipeline boundaries. Hashing an identifier is pseudonymization, not guaranteed anonymization: stable hashes can still enable tracking or dictionary attacks.

04. RED, USE, and the four golden signals

Start at the user-visible boundary, then descend through resource and dependency evidence.

Utilization is the busy fraction. Saturation is queued or delayed demand that cannot immediately use the resource. A database connection pool at 95% utilization may be healthy if waits remain near zero; a pool at 70% can be saturated if a few long transactions hold connections and waiters age. Pair every utilization graph with queue length or wait time and error evidence.

For offline pipelines, monitor input rate, successful completion rate, oldest item age, queue depth, retry rate, poison records, correctness checks, and estimated time to drain. A consumer can report high throughput yet fall behind when production is higher. For scheduled jobs, record last success time, duration, processed count, and expected next deadline.

05. SLIs, SLOs, SLAs, and error budgets

Reliability becomes actionable only when the good outcome, eligible population, target, and time window are explicit.

Event-based SLI

availability_SLI = good_responses / eligible_responses

good response:
  correct terminal response within 1 second

eligible population:
  syntactically valid, authenticated POST /orders requests

exclude only:
  predeclared cases outside the service promise, never incidents after they occur

Measure as close to the user's experience as practical, such as a trusted edge or end-to-end probe. Server metrics can miss DNS, TLS, routing, and response-delivery failures. Client telemetry adds coverage but can be delayed, blocked, duplicated, or attacker-controlled. Reconcile several views and name the authoritative SLI source.

Window choices

• Rolling window: every moment covers the preceding duration. It avoids a reset boundary but changes continuously.
• Calendar window: aligns with reporting or billing, but behavior near month boundaries can be misleading.
• Request-based budget: weights every eligible request equally.
• Time-based budget: counts good time slices and can hide severe low-volume failures.

Error-budget math

allowed bad fraction = 1 - 0.999 = 0.001

If requests in window = 120,000,000:
allowed bad requests = 120,000,000 * 0.001 = 120,000

Time intuition only, if traffic and impact were uniform:
28 days * 24 * 60 * 0.001 = 40.32 minutes

Request-based objectives do not literally grant a fixed number of outage minutes. A five-minute failure during peak traffic consumes more budget than one during quiet traffic. Error budgets are decision tools: when healthy, teams may accept measured release risk; when rapidly depleted, teams prioritize reliability, reduce change risk, or stop risky launches under a pre-agreed policy.

06. Availability math and dependency objectives

End-to-end reliability depends on topology, correlated failures, traffic, fallback behavior, and the exact definition of success.

Independence is often false. Replicas may share power, network, identity, configuration, release, certificate, storage, or operator error. Measure common-mode failure and place capacity across real failure domains. A service should not promise an objective that its critical serial dependencies make mathematically implausible. Negotiate dependency objectives, timeouts, support, maintenance, and overload behavior, then preserve end-to-end budget for application work.

A dependency can be "available" by its own status code SLI yet unusable to the caller because its p99 latency exceeds the caller's remaining deadline. Align semantic success, latency thresholds, regions, traffic classes, and measurement points. Record whether degraded fallback counts as good, separately good, or bad before incidents occur.

07. Burn-rate alerts

A burn rate measures how quickly the service consumes error budget relative to the sustainable rate.

burn_rate = observed_bad_fraction / allowed_bad_fraction

For a 99.9% SLO, allowed_bad_fraction = 0.001

If 1.4% of requests are bad:
burn_rate = 0.014 / 0.001 = 14

At a sustained 14x burn, a 28-day budget is consumed in:
28 / 14 = 2 days

A short window detects severe new failures quickly but is noisy. A long window is stable but slow. Multi-window, multi-burn-rate alerting combines them. A page might require both a high burn in a one-hour window and confirmation in a five-minute window. A lower-burn ticket can use a six-hour and thirty-minute pair. Exact thresholds follow the objective, response speed, traffic volume, and team's tested ability to act.

08. Tail latency and trustworthy percentiles

Averages hide the users who wait longest and the resources held by slow requests.

p99 is the value at or below which 99% of observations fall. It is not the slowest request and it says nothing about how bad the remaining 1% is. At 10,000 requests per second, 1% means 100 slow requests every second. Report p50, p95, p99, and sometimes p99.9 with sample count and maximum or tail buckets. Segment by bounded route, outcome, region, and workload class where it leads to an action.

Histogram design

• Place bucket boundaries around SLO thresholds and known latency modes.
• Use the same mergeable boundaries across instances when aggregating classic histograms.
• Keep enough resolution near the threshold. A 100 to 500 ms bucket cannot resolve a 250 ms SLO.
• Do not average instance p99 values. Aggregate distributions, then calculate a fleet percentile.
• Ensure the query window has enough observations. p99 from 20 requests is unstable.
• Track client and server duration separately to reveal network and queueing gaps.

Fan-out amplifies tails

If an operation waits for all 50 independent subrequests and each has a 1% probability of being slow, the probability that at least one is slow is 1 - 0.99^50, about 39.5%. Independence is only an approximation, but the lesson holds: wide fan-out makes component tail behavior become common end-to-end behavior. Reduce fan-out, batch, cache, use partial results where semantically safe, set deadlines, and observe the slowest child span.

09. Saturation, queueing, and Little's Law

Queueing delay often rises sharply before a resource reaches a visible 100% average.

For a stable system over a representative interval, Little's Law relates average in-system work L, average arrival or completion rate lambda, and average time in system W:

L = lambda * W

If 800 orders/second complete and mean end-to-end time is 0.25 second:
average in-flight orders = 800 * 0.25 = 200

If in-flight rises to 800 while throughput remains 800/second:
implied mean time = 800 / 800 = 1 second

The relationship is an accounting identity under stable boundaries and averages. It does not predict p99, prove independence, or make an unstable growing queue safe. Define whether "system" includes gateway wait, application processing, remote calls, or broker delay. Arrival rate and completion rate diverge while the queue grows, so use the correct long-run stable rate.

Queues hide overload temporarily

A queue absorbs a finite burst but adds no sustained service capacity. If arrivals are 1,200 per second and service capacity is 1,000 per second, backlog grows by 200 every second. After one minute, 12,000 requests wait. At 1,000 completions per second, the newest request already has about 12 seconds of queueing ahead of it, even before processing. Bound count, bytes, age, tenant share, and priority. Reject early when the result cannot meet its deadline.

Bottleneck capacity

End-to-end sustainable throughput is limited by the tightest required resource after accounting for per-request demand. If one order requires 2 database operations and the database safely sustains 6,000 operations per second, that dependency caps the path near 3,000 orders per second before replication work, background traffic, or failure reserve. Adding application instances cannot move that bottleneck and can exhaust database connections faster.

10. Capacity models and headroom

A capacity plan connects business demand to per-request resource demand, failure reserve, lead time, and a tested operating limit.

1. Model workload shape

• Current average, p95 interval, daily peak, event peak, and peak-to-average ratio.
• Reads, writes, payload bytes, fan-out, cache hit ratio, tenant skew, and expensive routes.
• Steady, burst, seasonal, launch, replay, recovery, and background workloads.
• Regions and zones, dependency limits, connection counts, data growth, and retention.
• Correctness and latency objectives that define the capacity boundary.

2. Translate demand into resource work

Forecast peak = 4,800 order requests/second
CPU service demand = 3.5 CPU-ms/request
Memory while active = 160 KiB/request
Mean active duration at target = 180 ms
Database operations = 2.4/request
Outbound bytes = 3.2 KiB/request

CPU cores busy = 4,800 * 0.0035 = 16.8 cores
Mean active requests = 4,800 * 0.180 = 864
Active-request memory = 864 * 160 KiB = 135 MiB
Database demand = 4,800 * 2.4 = 11,520 operations/second
Outbound bandwidth = 4,800 * 3.2 KiB = about 15 MiB/second

These are service-demand floors, not deployable capacity.

Add runtime, caches, buffers, telemetry, operating system, connection pools, background jobs, replication, retries, and variance. CPU demand can be calibrated with profile samples and load tests. Memory must cover live data, allocator behavior, runtime overhead, and safe garbage collection. Network sizing includes protocol overhead and both directions.

3. Size for failure and recovery

Suppose three zones each carry one-third of a 4,800 RPS peak. During one-zone loss, two zones each must accept 2,400 RPS. If an instance's tested safe capacity is 300 RPS at the required p99, each surviving zone needs at least 8 healthy instances, plus allowance for rollout, warm-up, instance failure, and forecast error. Ten per zone may be a reasoned starting point. "Thirty instances normally" is not the same calculation because placement determines failure survival.

4. State a headroom policy

Headroom can be defined as spare tested capacity divided by predicted load. It must cover forecast error, demand variance, autoscaling delay, cold cache, garbage collection, failover, draining, and repair lead time. A rapidly scalable stateless tier may need less static reserve than a database shard that takes weeks to split. The policy should name the failure scenario and time to add capacity, not apply one percentage everywhere.

5. Keep a capacity constraint table

11. Forecasting, uncertainty, and cost

Forecast ranges and decision dates are more useful than one precise-looking number.

1. Clean historical demand without erasing real incidents or launches.
2. Separate growth, weekly pattern, seasonality, scheduled events, and product changes.
3. Forecast per workload and resource driver, not only total requests.
4. Produce expected, high, and stress scenarios with documented assumptions.
5. Convert the forecast into an exhaustion date using the tested safe limit and reserve.
6. Subtract procurement, migration, validation, rollout, and rollback lead time to get a decision date.
7. Compare predictions with actuals and recalibrate demand coefficients regularly.

Linear extrapolation works only while product behavior and system efficiency remain stable. A new search feature can increase database work per request even when RPS is flat. Track unit economics: CPU-seconds, storage byte-months, egress bytes, telemetry bytes, database operations, and cost per successful business operation. Optimize the real constraint and include engineering complexity, reliability risk, and recovery time in cost decisions.

12. Load, stress, soak, scalability, chaos, and recovery tests

A valid performance experiment

1. State assumptions, SLO, hypothesis, workload distribution, and success criteria.
2. Use realistic payloads, key skew, cache state, fan-out, connections, and think time.
3. Generate load independently from the system under test and verify generator capacity.
4. Ramp gradually, hold steady long enough, and tag every run and service version.
5. Measure at the client and every likely bottleneck. Correct for coordinated omission.
6. Validate response correctness. Fast errors are not successful throughput.
7. Repeat, report variance and confidence, and preserve raw results and configuration.

Coordinated omission occurs when a load generator waits for one slow response before scheduling later requests that should have arrived. It undercounts latency during stalls. An open workload model schedules arrivals independently; a closed model maintains a fixed number of users or in-flight requests. Choose the model that matches reality and report it.

Chaos experiment template

Hypothesis:
  Losing one zone at forecast peak keeps order availability above 99.9%
  and p99 below 1 second.

Preconditions:
  backups and rollback verified; no active incident; observers assigned;
  surviving zones below tested failure limit.

Injection:
  remove order instances in one zone from service for 15 minutes.

Observe:
  edge SLI, routing, saturation, queue age, database load, retries,
  collector loss, autoscaling, and user-visible correctness.

Abort:
  budget burn exceeds threshold, database safety limit crosses,
  telemetry is lost, or blast radius escapes the named tenant cohort.

Recovery proof:
  capacity warms before traffic, backlog drains without retry storm,
  no duplicate orders, and SLI remains stable for 30 minutes.

13. Dashboards and actionable alerts

A dashboard supports a decision. An alert demands a specific human response.

Dashboard layers

1. Service overview: SLO attainment, current burn, traffic, tail latency, errors, saturation, deployments, and regions.
2. Dependency view: caller-observed rate, duration, errors, deadlines, retries, and breaker or limit state.
3. Resource view: CPU, memory, garbage collection, connections, queue wait, disk, network, and per-instance skew.
4. Capacity view: forecast, safe limits, failure reserve, lead time, cost per good operation, and backlog drain time.

Put changes on every graph: deployments, configuration, feature flags, traffic shifts, schema migrations, scaling, and dependency incidents. Use consistent units, time zones, color meaning, and y-axis behavior. A dashboard should link from symptom to likely cause while retaining an easy path back to user impact.

Alert contract

Every page should state:

• user or safety impact and the SLO at risk;
• affected service, operation, region, tenant scope, and start time;
• current value, threshold, burn rate, and supporting dashboard;
• first safe actions, runbook, escalation owner, and known automation;
• deduplication key and resolution condition.

Page on symptoms that require urgent human action. Ticket causes that need planned repair. Avoid paging on CPU alone when automation handles it and users are healthy. Avoid one alert per instance during a fleet event; group by service and failure domain while preserving scope. Test pages in drills, review false positives and missed incidents, and remove obsolete alerts after architecture changes.

14. Runbooks and on-call readiness

On-call is a supported operational role with bounded load, trained authority, and reliable tools.

A useful runbook

1. Purpose, service owner, dependencies, architecture, and trust boundaries.
2. Alert meaning, likely impact, and checks that confirm or disprove it.
3. Read-only diagnostic commands and dashboards first.
4. Mitigations ordered by safety, reversibility, and expected time to effect.
5. Prerequisites, exact scope, verification, rollback, and audit requirements for every mutation.
6. Escalation paths for application, platform, data, security, legal, and communications.
7. Recovery validation, backlog handling, and post-incident tasks.

Runbooks are executable knowledge. Assign an owner and review date, test them in staging and game days, and update them after topology or access changes. A command that says "restart everything" without scope, consequences, and verification is not a safe runbook.

On-call readiness checklist

• Primary and secondary rotations have sustainable staffing, handoff, and escalation coverage.
• Responders can access dashboards and audited emergency controls without sharing credentials.
• New responders shadow, lead a supervised incident, and pass game-day scenarios.
• Every critical service has ownership, dependency contacts, severity rules, and communication templates.
• Pager load, sleep interruption, response time, toil, burnout risk, and follow-up completion are reviewed.
• Monitoring and paging paths are tested independently of the production system they observe.

15. Incident command, severity, and communication

During a serious incident, separate coordination, technical work, and communication so responders can think and act safely.

Severity model

Define severities before incidents using user, financial, legal, security, data-integrity, region, duration, and error-budget criteria. Severity reflects impact and risk, not how difficult the bug looks. Start high when uncertainty and potential harm are high; downgrade with evidence.

Incident roles

• Incident commander: owns priorities, roles, decision cadence, escalation, and declaring recovery.
• Operations lead: directs technical investigation and mitigation workstreams.
• Communications lead: sends accurate, audience-appropriate updates on schedule.
• Scribe: records timestamps, evidence, hypotheses, decisions, changes, owners, and outcomes.
• Subject experts: investigate bounded questions and report concise evidence to operations lead.

Response state machine

DETECTED
  -> TRIAGED: validate signal, user impact, scope, severity
  -> MOBILIZED: assign commander, roles, channel, cadence
  -> MITIGATING: stop harm with safest reversible action
  -> RECOVERING: restore dependencies, capacity, and backlog gradually
  -> MONITORING: prove SLI and correctness are stable
  -> RESOLVED: close active response and preserve evidence
  -> LEARNING: postmortem, owned actions, trend analysis, verification

First fifteen minutes

1. Acknowledge, verify the page from an independent signal, and assess actual user impact.
2. Open one incident record and channel; state severity, scope, commander, and next update time.
3. Freeze risky changes if they could expand impact. Preserve security and audit evidence.
4. Compare current behavior with recent deployments, configuration, traffic, and dependency events.
5. Choose a reversible mitigation based on evidence, with an owner and expected observation time.
6. Escalate early when data integrity, security, legal duties, or multiple teams are involved.

Communication template

10:20 UTC - SEV-2 - Order creation latency

Impact: About 18% of order-create requests in ap-south exceed 1 second.
Start: 09:54 UTC. Existing orders and payment records remain readable.
Evidence: Edge p99 is 3.8 s; inventory calls account for most added time.
Action: New recommendation fan-out disabled at 10:17 UTC; rollback is continuing.
Risk: Queue age is falling; no duplicate or lost orders found in reconciliation sample.
Next update: 10:35 UTC, or sooner if impact changes materially.

Do not state an unverified root cause. Separate facts, hypotheses, actions, and results. Public updates should describe customer effect and progress without exposing credentials, exploitable details, personal data, or private customer names. Security incidents follow evidence handling, legal, privacy, and disclosure procedures in addition to reliability response.

16. Complete high-latency investigation

This case follows evidence across an edge, order API, inventory service, database, message system, and telemetry pipeline without jumping from correlation to cause.

System and assumptions

• Domain: an order platform with a 99.9% availability SLO and 99% under 1 second.
• Traffic: 4,000 RPS normal peak, three zones, synchronous inventory reservation.
• Optional recommendation lookup should have a 120 ms deadline and safe empty fallback.
• Order commit and outbox insert are one local transaction; notifications are asynchronous.
• One stable order idempotency key prevents duplicate effects during client retries.

1. Detect and scope

At 09:58, the one-hour SLO alert and five-minute confirmation page. Edge p99 rose from 420 ms to 3.8 seconds; 18% exceed the threshold. Error rate rose only slightly. The incident is latency-led, so waiting for 5xx errors would miss it. Region and route panels show only POST /orders in ap-south. Reads and other regions are normal.

2. Overlay changes

A new order API version reached 50% traffic at 09:51. Traffic volume is normal. The feature event stream shows recommendation enrichment enabled at 09:50. This temporal correlation makes the release a strong hypothesis, not proof.

3. Compare traces

Edge request                              3,820 ms
  Order API queue                          610 ms
  Authentication                           18 ms
  Recommendation lookup                   805 ms  deadline expected: 120 ms
  Inventory reservation attempt 1         790 ms  timeout
  Inventory reservation attempt 2       1,410 ms  success
    Inventory connection-pool wait      1,120 ms
    SQL update                              46 ms
  Order transaction                         72 ms
  Response                                  34 ms

Traces show that the new optional call ignores its intended deadline. It occupies order worker slots. Order queueing then delays inventory calls. A hidden client retry doubles inventory demand. The trace demonstrates a mechanism, but sampled traces alone do not quantify fleet impact.

4. Test the mechanism with metrics

• Order in-flight requests rose 3.4 times while useful throughput remained flat.
• Order queue wait rose before inventory latency, placing the first pressure in the caller.
• Inventory attempt rate is 1.7 times order rate, while original logical request rate is unchanged.
• Inventory CPU is 42%, but its database pool is full and waiter p99 exceeds 1 second.
• Database CPU and query execution remain normal; connection count reaches its safety cap.
• One zone receives 46% of traffic because slow connections distort least-connections routing.

This disproves "database CPU overload" and supports slot and connection retention plus retry amplification. A USE view catches saturation even though CPU utilization is modest.

5. Use logs and profiles

Structured inventory logs confirm wait-before-execute, not slow SQL. Order logs show the recommendation timeout field defaulted to zero for the new code path, meaning no application deadline. A profile comparison shows more parked threads awaiting network futures, not a CPU hot loop. Collector self-metrics show no relevant export drops, so absence of error logs is meaningful enough for this hypothesis.

6. Mitigate in dependency order

1. Disable optional recommendation enrichment with an audited feature control.
2. Disable the hidden inventory retry at the order client for the affected operation.
3. Reduce admission concurrency to stop new work from extending queues.
4. Roll back the new order version gradually while watching the edge SLI and pool wait.
5. Do not increase database connections blindly: the database is healthy and a larger pool could transfer overload to it.

7. Verify recovery

Queue age falls first, then connection wait, then p99. New traces contain no recommendation span and one inventory attempt. The team holds traffic steady for 30 minutes, checks each zone, reconciles idempotency outcomes against committed orders, confirms notification backlog drains, and checks telemetry loss counters. The commander declares recovery only after user SLI and correctness are stable.

8. Root cause and contributing conditions

Root cause: the new optional enrichment path failed to apply its 120 ms deadline and retained order concurrency slots during slow responses. Contributing factors: a client library enabled one retry, routing skewed under long-lived connections, staging tests used a fast recommendation stub, and the release dashboard lacked attempt-to-logical-request ratio. The incident was not caused by one person "forgetting a timeout"; reviews, defaults, tests, dashboards, and rollout controls all allowed the condition to reach users.

17. Root-cause analysis and blameless postmortems

A postmortem explains how normal system and organizational conditions produced the outcome, then creates verified changes that reduce recurrence or impact.

Postmortem structure

1. Summary, severity, owners, dates, duration, and affected objectives.
2. Quantified user, financial, data, security, and internal impact.
3. Detection method, response timing, and what delayed detection or mitigation.
4. Timestamped timeline with sources and uncertainty.
5. Trigger, root mechanism, contributing conditions, and why defenses did not stop it.
6. What worked, what did not, and where responders lacked information or authority.
7. Recovery proof and any remaining risk.
8. Action items with priority, owner, due date, tracking link, and verification method.

"Human error" is not a sufficient root cause. Ask why the action was reasonable with the information, interface, incentives, workload, and controls available. Blameless does not mean consequence-free or vague. It means separating learning from punishment while retaining clear accountability for controls and follow-up.

Causal analysis without a single-cause story

Five Whys can reveal deeper controls but can falsely imply one chain. Distributed incidents often form a graph: traffic shift plus retry policy plus stale capacity model plus missing alert. Use a fault tree, causal graph, or timeline to represent parallel conditions. Distinguish trigger from latent weakness and root mechanism from impact amplifier.

Strong follow-up actions

18. Production architecture and deployment practices

• Instrument shared protocol libraries, then add domain events at application boundaries.
• Version telemetry schemas and preserve compatibility during rolling deployments.
• Attach service name, version, environment, region, zone, and instance as resource identity.
• Deploy collectors across failure domains and protect them with memory limits and bounded queues.
• Keep the paging path independent enough to report a failure of the primary platform.
• Use synthetic probes from relevant networks for DNS, TLS, routing, and critical user journeys.
• Canary on user SLI, error-budget burn, resource demand, and telemetry health, not only process errors.
• Make every deployment, flag, config, migration, autoscaling, and failover change queryable as an event.
• Own dashboards, alerts, SLOs, capacity constraints, and runbooks in the same change lifecycle as code.

When observability fails

Instrument the pipeline itself with received, refused, queued, retried, exported, dropped, and processing-latency signals. Run an end-to-end canary record with known identity. Compare expected application volume with received backend volume. If collectors cannot export, keep bounded local evidence when policy permits, alert through an independent channel, and degrade expensive telemetry before business traffic. Sampling changes during an incident must be recorded because they alter interpretation.

19. Security, privacy, abuse, and trust boundaries

Telemetry often contains the most concentrated map of a system. Protect it as production data.

• Classify fields and prohibit credentials, session tokens, private keys, payment data, and raw request bodies by default.
• Use allowlists and typed structured fields. Redact before export and test redaction with generated sensitive fixtures.
• Authenticate and encrypt telemetry transport; authorize writes separately from queries and administration.
• Apply least privilege by team, service, tenant, environment, and purpose. Audit searches and exports.
• Treat incoming trace context and baggage as untrusted. Validate lengths and formats, and never use them for authorization.
• Prevent log injection by structured encoding and control-character handling.
• Rate limit tenant-controlled attributes and reject oversized headers to stop cardinality and storage abuse.
• Separate security audit logs from debug sampling, with tamper evidence and required retention.
• Protect incident channels, recordings, heap dumps, profiles, and postmortems, which can expose data and vulnerabilities.
• Define deletion, residency, legal hold, breach response, and backup policies for every telemetry store.

Trace IDs are identifiers, not secrets and not proof of identity. An attacker who guesses or supplies one must not retrieve its trace. Similarly, a tenant label must be derived from trusted authentication context, not a caller header. Keep user-facing incident communication accurate while coordinating security disclosure through the authorized process.

20. Performance, scalability, and telemetry cost

Establish an observability budget per good business operation and measure SDK CPU, allocation, export bytes, collector queue, backend ingest, query latency, and retention cost. Benchmark with instrumentation on. Never remove the only evidence for a critical invariant to save a small percentage without replacing it with a cheaper reliable signal.

21. Failure scenarios and safer corrections

22. Testing strategy

Unit tests

• Metric names, units, bounded labels, status classification, and zero initialization.
• Trace context validation, propagation, retry spans, and cancellation behavior.
• Redaction and allowlist rules against secrets, Unicode, encoded values, and nested objects.
• SLI eligibility and good-event classification at exact latency and status boundaries.
• Burn-rate calculations, missing data, counter resets, and low-volume cases.

Integration and contract tests

• Send one request across HTTP, message, and database boundaries; assert causal IDs and span relationships.
• Verify rolling deployment versions produce compatible telemetry and dashboards.
• Stop the collector or backend; prove bounded application overhead and observable export loss.
• Query a synthetic canary from the final store and validate end-to-end fields and redaction.
• Exercise alert rules on recorded fixtures and verify grouping, routing, runbook links, and resolution.

Load, fault, and recovery tests

• Measure instrumentation overhead and pipeline capacity at expected and stress traffic.
• Inject slow dependency, retry amplification, one-zone loss, clock skew, queue backlog, and exporter failure.
• Prove pages arrive within the required detection time and identify user impact.
• Execute the runbook with a responder who did not author it.
• Restore service, drain backlog, reconcile durable effects, and verify SLI plus pipeline stability.

23. Hands-on exercises and expected reasoning

Exercise 1: Design an order-service SLO

Define availability and latency objectives for order creation, including invalid requests, idempotent duplicates, client cancellation, and accepted asynchronous responses.

Expected reasoning: Choose the user-visible edge as the main measurement point; define valid authenticated attempts as eligible; distinguish a safely replayed idempotent result from a duplicate effect; set a latency threshold from user need; specify rolling window and traffic segments; name exclusions before incidents; preserve a separate correctness invariant for no double charge.

Exercise 2: Capacity for zone loss

Peak forecast is 9,000 RPS across three zones. One instance safely handles 250 RPS at the SLO. Instances need 8 minutes to warm. Size the minimum failure capacity and explain reserve.

Expected reasoning: After one-zone loss, two zones each carry 4,500 RPS, requiring 18 ready instances per surviving zone. That is a mathematical floor. Add reserve for instance failure, forecast error, rollout, warm-up, and skew. Pre-provision enough capacity to bridge eight minutes because autoscaling cannot help immediately. Verify the database and other dependencies at 9,000 RPS under failure, not only the stateless tier.

Exercise 3: Explain misleading evidence

Stored traces show 30% errors, while authoritative metrics show 0.6%. Tail sampling retains all errors and 1% of successes. Which is correct?

Expected reasoning: The trace store is intentionally biased and its raw fraction does not estimate fleet error rate. Use the pre-sampling metric for the SLI. Traces explain example failures. If known inclusion probabilities and weights are available, statistical estimation may be possible, but the naive ratio remains wrong.

Exercise 4: Incident simulation

Run a game day where notification consumer throughput drops below production, backlog age grows, and customer-facing requests remain healthy.

Expected reasoning: Page only if the oldest item threatens the business deadline; otherwise create an actionable ticket. Identify consumer or dependency saturation, stop poison retries, add safe capacity, predict drain time from net rate, protect ordered partitions, and verify no message loss or duplicate external send. Update capacity and runbook from results.

24. Interview questions and model answers

1. What is the difference between monitoring and observability?

Monitoring checks known behavior using predefined signals and conditions. Observability is the practical ability to investigate internal behavior from emitted evidence, including failures not anticipated when alerts were written. Monitoring is a use of observability data. Neither is a product checkbox; instrumentation quality, context, pipeline reliability, queryability, and team practice determine the outcome.

Follow-up: Can a system be observable with logs only?

Possibly at small scale if logs preserve complete causal and resource evidence, but distributed scale makes this expensive and statistically weak. Metrics, traces, profiles, and events provide complementary compression and correlation.

2. How would you define an SLO for checkout?

Start from the user journey and define eligible attempts, correct results, latency threshold, measurement point, window, target, and segmentation. Include correctness, such as at most one charge, separately from availability. Decide how cancellations, invalid input, dependency errors, and accepted asynchronous states count before incidents. Use an objective stricter than the contractual SLA when operational margin is needed.

Follow-up: Why not use server uptime?

A process can be running while DNS, TLS, routing, queues, dependencies, or correctness fail for users. User-visible good events better represent the service promise.

3. Explain burn rate.

Burn rate is observed bad fraction divided by the SLO's allowed bad fraction. A rate of 1 consumes budget exactly at the sustainable pace; 10 consumes it ten times faster. Pair a stable long window with a responsive short window so pages are both significant and timely.

Follow-up: Why not page whenever error rate exceeds 1%?

The same rate has different significance under different SLOs, windows, traffic, and duration. Burn rate connects impact directly to the reliability promise and remaining budget.

4. Why is p99 hard to aggregate?

A percentile is a position in a distribution, not an additive value. Averaging per-instance p99 weights instances incorrectly and loses distribution shape. Merge histogram counts with compatible buckets or another mergeable distribution, then calculate the fleet percentile. Include count and bucket resolution.

Follow-up: What does p99 of 800 ms mean?

In the measured population and window, 99% completed at or below about 800 ms. It does not mean every user is below 800 ms or that the slowest 1% is near 800 ms.

5. Apply Little's Law to a service.

In stable conditions, average in-system work equals average throughput times average time in the defined system. At 2,000 requests per second and 200 ms mean time, expect about 400 in flight. A rising in-flight count at flat throughput implies increasing time or an unstable queue. The law uses averages and does not directly predict tails.

Follow-up: Can it size a connection pool exactly?

It supplies a starting concurrency estimate for the portion using connections. Variance, transactions, dependency limits, timeouts, failure reserve, and measured wait still determine the safe pool.

6. How do you avoid telemetry cardinality explosions?

Use low-cardinality bounded labels for metrics, normalize routes, place request-level identity in controlled logs or traces, estimate combinations before launch, enforce SDK and backend limits, and alert on overflow. Review tenant-controlled fields as an abuse boundary.

Follow-up: Is hashing a user ID enough?

No. It remains high cardinality and often remains linkable personal data. Hashing changes the representation, not the fundamental scale or privacy issue.

7. What makes an alert actionable?

It identifies urgent user or safety impact, affected scope, current evidence, expected responder, first safe action, runbook, escalation, and resolution condition. If nobody should act now, route it to a ticket or dashboard instead of a pager.

Follow-up: Should high CPU page?

Only if it represents urgent risk requiring a human and automation cannot safely handle it. SLO burn and saturation usually provide better paging context.

8. How do you capacity-plan a distributed service?

Forecast workload shape, translate each operation into resource demand, identify the bottleneck, benchmark its safe limit under the SLO, model zone and dependency failures, add reasoned headroom, include scaling and procurement lead time, and compare forecast with actuals. Repeat per resource because CPU, database, storage, network, and backlog can exhaust on different dates.

Follow-up: What is the biggest modeling mistake?

Treating average RPS and a stateless benchmark as capacity while ignoring workload mix, skew, dependencies, recovery traffic, and failure placement.

9. What do you do first in a high-latency incident?

Verify user impact, establish severity and ownership, preserve a timeline, compare SLI dimensions and recent changes, and choose the safest reversible mitigation. Investigate queue wait, saturation, dependency attempts, and traces before simply adding capacity or restarting. Keep communications factual and timestamped.

Follow-up: When is rollback unsafe?

When data or protocol changes are not backward compatible, rollback crosses a migration boundary, the old version has a security flaw, or the suspected change is not actually isolated. Evaluate compatibility and use a scoped feature disable or traffic shift when safer.

10. What makes a postmortem blameless and useful?

It explains impact, evidence, timeline, trigger, mechanisms, contributing conditions, response, and recovery without stopping at individual fault. It assigns concrete preventive, detective, mitigating, and recovery actions with owners and verification. Blamelessness enables honest learning; accountability ensures improvements are completed.

Follow-up: How do you prevent action-item decay?

Track items in the normal work system, prioritize by risk, assign one owner and date, review them regularly, escalate overdue critical work, and close only after the verification evidence exists.

25. Revision cheat sheet

• Monitoring checks known conditions; observability supports new questions from emitted evidence.
• Metrics aggregate, logs record facts, traces show causal paths, profiles show resource hot spots, and events show changes.
• Use stable structured fields and correlate with trace and span identifiers.
• Validate W3C trace context and treat baggage as untrusted, non-authoritative data.
• The telemetry pipeline needs bounded buffers, loss metrics, security, and capacity.
• Head sampling decides early; tail sampling uses completed-trace evidence but costs memory and delay.
• Never calculate fleet error rate from biased tail-sampled traces.
• Metric cardinality multiplies across labels; keep unbounded IDs out of metrics.
• RED means rate, errors, duration. USE means utilization, saturation, errors.
• Golden signals are latency, traffic, errors, and saturation.
• An SLI measures behavior, an SLO sets a target, an SLA is an agreement, and an error budget is allowed not-good service.
• Define good event, eligible population, measurement point, target, and window.
• Serial dependency availability multiplies only under the stated simplified assumptions.
• Burn rate equals observed bad fraction divided by allowed bad fraction.
• Use multi-window alerts to combine significance with fast detection.
• Aggregate distributions before calculating a fleet percentile. Never average p99 values.
• Wide fan-out turns rare component tail latency into common request latency.
• Little's Law is L = lambda * W for stable averages and defined boundaries.
• A queue absorbs a finite burst but adds no sustained capacity.
• Capacity is sustainable good work under latency, correctness, and failure assumptions.
• Forecast workload mix and resource demand, then include failure reserve and lead time.
• Load tests validate expected demand; stress finds the knee; soak finds time effects.
• Chaos tests require a hypothesis, bounded blast radius, abort rule, and recovery proof.
• Page only when urgent human action can improve an important outcome.
• A runbook includes diagnosis, scoped mitigation, verification, rollback, and escalation.
• Incident command separates coordination, operations, communication, and record keeping.
• Mitigate harm before proving a complete root cause, using reversible evidence-driven changes.
• Declare recovery only after SLI, correctness, backlog, and telemetry are stable.
• Blameless postmortems analyze systems and still assign owned, verifiable actions.

26. Primary official references

• OpenTelemetry Specification: Overview - signal, provider, context, resource, baggage, and Collector concepts.
• OpenTelemetry signals - current official concepts for traces, metrics, logs, and baggage.
• OpenTelemetry sampling - head sampling, tail sampling, representativeness, and trade-offs.
• OpenTelemetry Collector - receive, process, batch, retry, filter, and export pipeline responsibilities.
• W3C Trace Context Recommendation - interoperable traceparent and tracestate formats and security rules.
• W3C Baggage Candidate Recommendation - interoperable propagation and privacy and security considerations for baggage.
• Prometheus instrumentation practices - online, offline, and batch signals, metric types, labels, and cardinality guidance.
• Prometheus histograms and summaries - aggregation, quantiles, buckets, and Apdex-style threshold calculations.
• Google SRE Book: Monitoring Distributed Systems - four golden signals, tail latency, and symptom-focused alerting.
• Google SRE Workbook: Implementing SLOs - SLI specification, measurement, objectives, and error budgets.
• Google SRE Workbook: Alerting on SLOs - burn rates and multi-window, multi-burn-rate alert design.
• Google SRE Book: Software Engineering in SRE - capacity planning, load balancing, and engineering operational work.
• Google SRE Workbook: Non-Abstract Large System Design - demand, resource, failure, and capacity reasoning for large services.
• Google SRE Workbook: On-Call - sustainable rotations, escalation, pager categories, and responder support.
• Google SRE Workbook: Incident Response - command roles, incident practices, coordination, and communications.
• Google SRE Workbook: Postmortem Culture - blameless learning, postmortem structure, and actionable follow-up.
• NIST SP 800-61 Revision 3 - current incident-response recommendations integrated with cybersecurity risk management.
• Oracle Java Flight Recorder API Programmer's Guide - continuous low-overhead runtime event recording and profiling configuration.