Scalability, Load Balancing, Service Discovery, and Rate Control - Complete Notes

A language-neutral guide to growing a service safely, spreading work across changing instances, finding healthy destinations, and controlling demand before overload becomes a cascading failure.

00. One capacity-control loop

Scaling supplies capacity, discovery publishes capacity, load balancing places work on capacity, and rate control keeps admitted work within capacity.

Treat these as one feedback system, not four unrelated features. A new instance is useful only after it starts, warms, becomes ready, appears in discovery, and receives an appropriate share of traffic. A terminating instance is safe only after it stops receiving new work, finishes or transfers existing work, disappears from discovery, and shuts down. Rate control protects this loop when demand changes faster than capacity can change.

observed demand and saturation
            |
            v
  capacity controller ---- creates or removes instances
            |                         |
            |                         v
            |             starting - warming - ready - draining - stopped
            |                         |
            v                         v
  admission and rate control -> discovery -> load balancer -> ready instance
            ^                                               |
            |                                               v
            +------ latency, errors, queue depth, load ------+

Scalability

The ability to handle a larger workload by adding resources without unacceptable loss of efficiency, correctness, reliability, or cost control.

Elasticity

The ability to add and remove capacity in response to changing demand. Scalability says growth is possible. Elasticity says the system adjusts over time.

Load balancing

Choosing a healthy destination for each connection, request, stream, job, or partition.

Service discovery

Mapping a stable service identity to its current reachable endpoints and relevant metadata.

Rate limiting

Enforcing a policy on work accepted per identity, operation, resource, or time interval.

Admission control

Deciding whether the system has enough budget to start a particular unit of work now.

Backpressure

A signal from a constrained consumer that causes upstream producers to slow, wait, or stop.

Load shedding

Rejecting or dropping selected work to preserve useful service when the system is near or beyond sustainable capacity.

01. Vertical and horizontal scaling

Vertical scaling makes one resource larger. Horizontal scaling adds independently schedulable resources.

Real designs combine both. A database may scale vertically until partitioning is justified while stateless API servers scale horizontally from the beginning. Horizontal scaling is not automatic linear speedup. If ten instances share one saturated database, lock, queue partition, network link, or global counter, the shared bottleneck still sets total throughput.

Scalability is not single-request performance

Performance describes behavior at a particular resource level and workload. Scalability describes how behavior changes as workload and resources change. An implementation can answer one request very quickly but scale poorly because all requests acquire a global lock. Another can have modest per-request latency yet scale well across many independent workers.

scaling efficiency =
  throughput with N resources
  -----------------------------------
  N * throughput with one resource

Example:
  1 instance  = 1,000 successful requests/second
  8 instances = 6,400 successful requests/second
  efficiency  = 6,400 / (8 * 1,000) = 80%

Measure successful useful work at the required latency and error target. Counting accepted requests, queued jobs, or failed attempts can make a collapsing system look productive.

02. Stateless, stateful, and shared-nothing services

Compute scales most easily when any ready instance can handle the next request and durable state has an explicit owner.

A stateless service instance does not require its own process memory or local disk from an earlier request to handle the next request. It may still use durable databases, caches, object stores, and queues. Stateless refers to the replaceability of the serving instance, not the absence of application data.

A stateful instance owns or temporarily holds information that affects routing or correctness, such as a game session, WebSocket connection, in-memory workflow, partition leader, local index shard, or unreplicated write buffer. It can scale horizontally, but placement, replication, ownership transfer, and recovery must be designed.

Shared-nothing means each worker owns independent CPU, memory, and a partition of data or work, with no shared memory or disk on the critical path. It reduces coordination and can scale well, but requests spanning partitions require fan-out or coordination. Shared-nothing does not mean no communication, no replication, or no shared control plane.

03. Elasticity and autoscaling as control theory

An autoscaler observes a delayed signal, calculates a desired capacity, performs a delayed action, and repeats. Poor signals or timing make it oscillate or react too late.

every evaluation interval:
  observed = stable_measurement_window(metric)
  ratio = observed / target
  desired = ceil(current_ready_capacity * ratio)
  desired = clamp(desired, minimum, maximum)
  desired = apply_scale_rate_limits(desired)
  desired = apply_stabilization_history(desired)
  request_capacity(desired)

This ratio form is used by common orchestration controllers, but the important design is platform-neutral. A controller needs a metric causally related to load, a safe target below the performance cliff, bounds, cooldown or stabilization, and separate scale-up and scale-down behavior.

Choose a signal that predicts required work

Prefer demand or concurrency signals when they lead saturation. Never blindly scale a caller on dependency latency. If the database is slow, adding API instances can increase database concurrency and worsen the incident. Cap fleet concurrency at the dependency's tested capacity.

Warm-up, cold starts, and reaction time

Capacity is not available when an instance object is created. Provisioning a machine, pulling an image, starting a runtime, compiling hot code, creating connection pools, loading models, fetching configuration, and warming caches all add delay. Measure time from the scaling decision until the endpoint serves production-shaped traffic within its SLO.

reaction time =
  metric collection delay
  + evaluation delay
  + scheduler and provisioning delay
  + application startup
  + readiness confirmation
  + discovery propagation
  + traffic warm-up

If burst rise time < reaction time:
  pre-provision, schedule capacity, queue bounded work, or shed load.

Keep a minimum warm fleet for latency-sensitive traffic. Scale up faster than scale down. Use a stabilization window so one quiet interval does not remove capacity that will soon be needed. During a large rollout, autoscaling must account for old and new capacity, readiness, and deployment surge.

Scale for failure, not only normal traffic

In a three-zone design that must survive one zone loss, each zone cannot normally run at full utilization. If traffic is balanced equally, two remaining zones must absorb the third zone's share. Also reserve capacity for uneven distribution, deployments, recovery work, and growth.

peak demand                       = 24,000 requests/second
tested safe capacity per instance = 1,500 requests/second
instances per zone                = 6
zones                             = 3

normal total capacity             = 27,000 requests/second
after one-zone loss               = 18,000 requests/second  FAIL

Use 9 instances per zone:
after one-zone loss               = 18 * 1,500 = 27,000 requests/second
failure headroom                  = 3,000 requests/second

04. Layer 4 and Layer 7 load balancing

A Layer 4 balancer understands connections and transport addresses. A Layer 7 balancer understands an application protocol such as HTTP.

Layer names describe responsibility, not a guarantee about implementation. A flow-hash Layer 4 balancer keeps all packets in one transport flow on one destination. HTTP/2 or a multiplexed RPC connection may carry many concurrent requests, so a few long-lived connections can produce uneven backend request load. A Layer 7 proxy can place requests individually, but only if it terminates or participates in the connection.

Client-side and server-side balancing

Server-side balancing

Clients call one stable virtual address or proxy. The proxy discovers endpoints and selects one. This centralizes policy and simplifies clients, but adds a hop and requires the proxy tier to be highly available and scalable.

Client-side balancing

Each client receives an endpoint set and selects directly. It can remove a hop, react to per-request state, and spread control, but every client library must handle discovery, connection pools, health, security, policy updates, and stale membership correctly.

A sidecar or node-local proxy is a hybrid. The application sees a simple local endpoint while the proxy performs client-side discovery and balancing. It standardizes behavior but adds another process, resource cost, configuration plane, and failure boundary.

05. Placement algorithms and their assumptions

A load-balancing algorithm observes an imperfect view of work. Choose the cheapest signal that correlates with the resource you need to equalize.

Why two random choices help

Pure random placement can accidentally add work to an already busy server. Sampling two servers and choosing the lower-load one sharply reduces the chance of building a very long tail while avoiding a global search for the absolute least-loaded server.

function choose(endpoints):
  a = random_ready_endpoint(endpoints)
  b = random_ready_endpoint_excluding(endpoints, a)
  if normalized_load(a) <= normalized_load(b):
    return a
  return b

normalized_load may be:
  active_requests / endpoint_capacity_weight

Do not use a metric that arrives slower than the requests it controls without smoothing. Independent balancers can all observe the same "least loaded" endpoint and stampede it. Random sampling and per-balancer views reduce synchronized choice.

Hashing and bounded disruption

A simple hash(key) mod N remaps most keys when N changes. Consistent hash rings, rendezvous hashing, and Maglev-style lookup tables aim to keep most mappings stable as endpoints join or leave. Virtual nodes or weights help represent unequal capacity.

Hashing is affinity, not health. The selection procedure still needs a policy when the preferred endpoint is unavailable. Replicated ownership, bounded fallback, and hot-key detection are essential. A single celebrity account can exceed one instance's capacity even when the average hash distribution is perfect.

06. Health, readiness, draining, and slow start

"The process exists," "the endpoint can serve," and "the endpoint should receive new traffic" are different states.

created
   |
   v
starting -- startup failed --> restart or terminal failure
   |
   v
warming -- not ready --> excluded from normal traffic
   |
   v
ready -- temporary dependency pressure --> not ready
   |  ^                                  |
   |  +---------- recovered -------------+
   |
scale-in, rollout, or shutdown
   v
draining -- deadline reached --> cancel or transfer remaining work
   |
   v
stopped

Liveness

Can the process make progress, or should the supervisor restart it? A liveness check should not fail merely because an optional dependency is slow.

Readiness

Should this endpoint receive new normal traffic now? It may include local initialization and ability to reach critical resources, but must avoid synchronized fleet-wide failure.

Startup

Has slow initialization completed enough for liveness and readiness evaluation to become meaningful?

Passive health

The balancer infers health from real connection failures, response codes, latency, or timeouts.

Active health

A checker sends synthetic probes at intervals and applies success and failure thresholds.

Health checks are sampled and delayed. With a 10-second interval and three failed probes required, traffic may continue for roughly 20 to 30 seconds after failure, plus propagation time. Aggressive checks consume capacity and can eject healthy-but-busy instances, shifting more traffic to the remainder. Use hysteresis: multiple failures to eject, multiple successes to restore, and jittered probe timing.

Graceful drain sequence

1. Mark the endpoint not ready or draining in the authoritative control plane.
2. Wait for load balancers and discovery consumers to stop selecting it.
3. Stop accepting new requests or connections at the application boundary.
4. Finish work that can complete before the termination deadline.
5. Signal clients to reconnect, checkpoint streams, or transfer partition ownership.
6. Cancel remaining work safely, close pools, flush bounded telemetry, then exit.

The sequence must match connection behavior. Removing an endpoint from DNS does not close existing TCP connections. A reverse proxy may keep an upstream pool open. HTTP/2 and WebSocket connections may last far longer than the DNS TTL. Use protocol-specific drain signals and a maximum connection age where appropriate.

Slow start after readiness

Readiness is often binary while capacity ramps gradually. A newly ready endpoint can have empty caches, cold runtime optimization, and new connection pools. Slow start gives it a small initial weight and raises the weight over a measured warm-up window. If every endpoint starts together, slow start cannot create a warm destination, so rolling deployment and minimum warm capacity still matter.

07. Connection reuse, sticky sessions, and uneven load

The balancer may distribute connections evenly while requests, bytes, CPU, or user impact remain badly skewed.

Connection pooling reduces handshake cost and latency but creates persistence. If each client opens only a few connections, a recently added endpoint receives little traffic until pools turn over. If one HTTP/2 connection carries thousands of requests, Layer 4 balancing sees one flow. Bound connection lifetime, refresh endpoint sets, and measure request-level load per endpoint.

Sticky sessions route an identity repeatedly to one endpoint, often using a cookie, source address, or hash. They can preserve a local cache or unavoidable in-memory session, but they reduce balancing freedom, complicate failover, and can concentrate heavy tenants. Client IP affinity is unreliable behind NAT, carrier networks, and forward proxies.

08. Service discovery and membership

Discovery answers, "What endpoints currently represent this logical service?" It does not prove that an endpoint will still be healthy when the next packet arrives.

Registry-based discovery

instance starts
  -> authenticates to registry
  -> registers identity, address, port, locality, version, and health metadata
  -> renews lease or heartbeat

client or proxy
  -> watches or polls service membership
  -> validates revision
  -> filters ready endpoints
  -> balances requests

instance drains or lease expires
  -> registry publishes removal
  -> consumers update local endpoint sets

A lease bounds stale membership after a crashed instance, but short leases increase control-plane load and false expiration during pauses or partitions. Long leases leave dead endpoints visible longer. Watches reduce polling delay but need reconnect, revision, compaction, and full-resync behavior. The data plane should continue using a last-known-good set during a brief registry outage, with an age limit and alert, instead of making every request depend synchronously on the registry.

DNS-based discovery

DNS can map a service name to virtual addresses, endpoint addresses, or SRV records containing target host, port, priority, and weight. It is broadly compatible and naturally cached. However, effective caching occurs in authoritative servers, recursive resolvers, operating systems, runtimes, applications, and connection pools. The configured TTL is not an instant failover guarantee.

• Use multiple records and assume clients implement selection differently.
• Keep old endpoints alive through the expected cache and connection drain interval.
• Distinguish negative caching from an empty healthy endpoint set.
• Refresh before cached data expires, and define last-known-good behavior.
• Do not publish endpoints before readiness or remove capacity faster than caches converge.
• Secure update authority and validate names to prevent traffic redirection.

Discovery failure matrix

09. Reverse proxies, API gateways, and global traffic

These components may all forward traffic, but their policy scope and trust boundaries differ.

Reverse proxy

Accepts traffic for one or more upstream applications, terminates or passes protocols, selects backends, and may add caching, compression, TLS, and observability.

API gateway

A policy-focused application entry point that may authenticate users, authorize routes, enforce quotas, transform versions, validate payload limits, and aggregate APIs. Business authorization still belongs near the owning service.

Global traffic manager

Chooses a region or site using DNS, anycast, global proxies, latency, health, residency, and capacity policy.

client
  -> global policy: region with residency, health, and spare capacity
  -> regional edge: authentication, coarse quota, route and payload policy
  -> zone-aware balancer: prefer local healthy capacity
  -> service endpoint: concurrency admission and fine-grained tenant limit
  -> dependency: its own admission limit and isolation

Locality-aware routing reduces latency and network cost, but local preference must respect capacity. Spillover should be gradual and bounded. A zone outage that instantly transfers all traffic can overload the surviving zones. Global failover must consider data availability, consistency, residency, dependency health, and failure capacity, not only whether an HTTP probe returns success.

Gateways are high-value attack targets and potential common failure points. Run multiple instances, minimize request-path dependencies, use bounded configuration rollout, preserve the original authenticated identity securely, remove untrusted forwarding headers, and avoid unbounded body buffering.

10. Rate limits, quotas, and admission

Rate is speed, quota is allocation over an interval, and concurrency is work already occupying capacity. A robust service often needs all three.

Choose a key aligned with the protected resource and fairness goal: authenticated account, tenant, API key, user, destination, operation, IP prefix, or a combination. IP alone can punish thousands of users behind one NAT and is easy for a distributed attacker to rotate. Authentication alone does not protect unauthenticated endpoints.

Apply cheap coarse controls at the edge and authoritative fine-grained controls near the resource. Edge limits reduce attack traffic and bandwidth. Service-local concurrency limits protect actual database pools, CPU, and downstream capacity even if traffic bypasses or is unevenly distributed across edge nodes.

11. Rate-control algorithms

Algorithms differ in burst behavior, memory, precision, distribution cost, and boundary effects.

Token bucket

Tokens arrive at rate r up to capacity b. A request costing c tokens is admitted only when enough tokens exist. The long-term rate is approximately r, while an idle client may burst up to b. Weighted costs can represent expensive operations.

function allow(bucket, now, cost):
  elapsed = max(0, now - bucket.last_refill)
  bucket.tokens = min(bucket.capacity,
                      bucket.tokens + elapsed * bucket.refill_per_second)
  bucket.last_refill = now

  if bucket.tokens < cost:
    return REJECT

  bucket.tokens = bucket.tokens - cost
  return ALLOW

Use a monotonic clock within one process. In a distributed implementation, do not trust client clocks. Keep atomic update semantics per key. Bucket capacity is an explicit burst promise, not an implementation detail.

Leaky bucket and paced queues

A leaky bucket drains at a fixed rate. It can mean a meter mathematically similar to token bucket, or a bounded queue that releases work at a steady pace. The queued form smooths bursts but adds latency and memory. Reject when the queue is full or when predicted wait exceeds the request deadline. Never create an unbounded queue to avoid saying no.

Fixed window counter

Count requests in aligned windows such as each minute. It is simple and storage-efficient, but a client can send a full minute's allowance at 12:00:59 and another at 12:01:00, producing nearly twice the intended load near the boundary.

window = floor(server_time / window_size)
key = (identity, operation, window)
count = atomic_increment(key, expiry = window_size + safety_margin)
allow if count <= limit

Sliding log and sliding counter

A sliding log stores timestamps and counts only events in (now - window, now]. It is accurate but costs memory and cleanup proportional to traffic. A sliding counter approximates the result using the current and previous fixed windows:

fraction_remaining_in_previous = 1 - elapsed_in_current / window_size
estimated =
  current_window_count
  + previous_window_count * fraction_remaining_in_previous

allow if estimated + request_cost <= limit

12. Distributed counters, fairness, and availability

A global limit is a distributed coordination problem. Exactness, latency, cost, and availability cannot all be maximized.

Implementation patterns

Suppose the global rate is 10,000 requests per second across 100 gateways. Giving each gateway 100 requests per second is safe only when traffic is even. One gateway receiving a large customer's traffic will reject while others waste budget. A central counter improves utilization but may become the new bottleneck. Leasing batches gives gateways temporary rights and amortizes coordination. Use small leases for precision and larger leases for efficiency.

Fairness and cost

Equal request counts are not fair when costs differ. A file export and metadata lookup may differ by thousands of CPU milliseconds and bytes. Use weighted units, separate pools, or concurrency budgets for expensive work. In multi-tenant services, combine:

• A global emergency ceiling that protects the service.
• A tenant rate and concurrency limit that prevents noisy-neighbor harm.
• A reserved or minimum share for critical tenants or operations.
• A borrowable shared pool for unused capacity, reclaimed when owners need it.
• Priority that cannot starve lower classes forever.

Return a stable machine-readable reason, limit scope, and retry guidance. HTTP APIs commonly use 429 Too Many Requests for policy limiting and may include Retry-After. Use 503 Service Unavailable for temporary service-wide inability when that distinction helps clients. Under severe attack, generating detailed rejection responses can itself be costly, so dropping traffic earlier may be safer.

Limiter dependency failure

Fail-open preserves availability but may expose a protected dependency or allow abuse. Fail-closed preserves the limit but turns a limiter outage into application unavailability. Choose per route: a public search endpoint may use a conservative local fallback, while an expensive money-moving operation may fail closed for policy and safety. Cache policy, use local emergency ceilings, and alert on fallback mode.

13. Backpressure, queues, and load shedding

Rate limits enforce planned policy. Load shedding responds to actual pressure. Backpressure carries the constraint upstream.

When arrival rate exceeds completion rate, queue length grows. Queues absorb short bursts, not sustained overload. Long queues raise tail latency, consume memory, hold connections, and cause clients to time out and retry while the server continues useless work.

estimated_wait =
  queued_work_units / recent_sustainable_completion_rate

if request.deadline_remaining <=
   estimated_wait + estimated_service_time + safety_margin:
  reject_without_starting_dependency_work

if in_flight_for_tenant >= tenant_concurrency_limit:
  reject_or_defer

otherwise:
  admit

Prefer concurrency limits for work whose duration varies. Adaptive concurrency controllers can change the limit based on latency or queueing, but must use floors, ceilings, smoothing, and canaries. A dependency's limit should be below its measured saturation point.

Define a shedding order before an incident

1. Stop speculative work such as prefetching and cache refresh.
2. Pause batch jobs, replays, analytics, and non-urgent synchronization.
3. Serve cheaper degraded results, smaller pages, or stale safe data.
4. Reject low-priority interactive traffic with clear retry behavior.
5. Preserve critical writes or safety operations within reserved capacity.
6. If collapse continues, sharply reduce admitted traffic, recover a warm fleet, then ramp.

Protect every layer because a gateway cannot see fan-out cost perfectly. A request admitted at the edge may cause 100 downstream calls. The downstream service must remain in control of its own concurrency. Propagate deadlines and cancellation so expired work stops consuming capacity.

14. Worked design: API across three zones

Scale an order-read and order-create API from one instance to a multi-zone fleet while protecting its database, payment dependency, and tenants.

Requirements and assumptions

• Peak external demand is 24,000 requests per second, 90% reads and 10% creates.
• Traffic can jump 3 times within 30 seconds during a sale.
• Read p99 target is 250 ms; create p99 target is 800 ms.
• One zone may fail without exceeding the steady-state latency target.
• Create requests require authenticated tenant identity and idempotency keys.
• The primary database safely supports 4,000 writes per second and 8,000 concurrent queries.
• One warm API instance safely serves 1,500 mixed requests per second at 60% CPU.
• A new instance takes 90 seconds from decision to fully warm capacity.

Architecture and request path

clients
  |
  v
global traffic policy
  |
  v
multi-zone L7 gateway fleet
  |  - authenticate identity
  |  - request size and route policy
  |  - coarse tenant token bucket
  |
  +---------+---------+
  |         |         |
zone A    zone B    zone C
  |         |         |
local load balancer prefers ready local API instances
  |         |         |
  +---------+---------+
            |
     API admission control
       |            |
       |            +-- create: tenant concurrency + idempotency
       |
       +-- read: bounded DB concurrency, optional safe degradation
            |
     database and dependencies with their own limits

Capacity and scaling policy

Run nine instances per zone, giving 40,500 requests per second normal tested capacity. After one zone fails, 18 instances provide 27,000 requests per second, or 12.5% headroom over normal peak. Validate that gateway, database, network, and connection pools also survive the failure.

• Minimum is nine ready instances per zone, never a global pool of 27 placed arbitrarily.
• Scale on the maximum of request rate per ready instance and normalized in-flight work.
• Schedule extra capacity before known sales because the 30-second burst outruns 90-second start.
• Scale up aggressively within provider and database ceilings, then apply slow start.
• Scale down at most 10% per five minutes after a 15-minute stable low-demand window.
• Hold capacity during zone impairment, rollout, or discovery lag.

Create-order admission flow

1. Gateway authenticates tenant and rejects oversized or malformed input.
2. Coarse bucket checks tenant request rate.
3. Zone balancer chooses two ready endpoints and sends to the less loaded.
4. API validates deadline and idempotency key.
5. Hierarchical admission checks:
     global create ceiling
     tenant weighted rate
     tenant create concurrency
     database write concurrency
6. If rejected, return a stable reason and retry guidance.
7. If admitted, reserve concurrency, call required dependencies once,
   persist result, release reservation, and record outcome.
8. A repeated idempotency key returns the recorded outcome, not a second order.

Zone-loss timeline

Why this design fits, and when it changes

The API fleet is stateless, latency-sensitive, and horizontally partitionable by request, so multi-zone active serving is suitable. The database remains a shared constraint, so concurrency admission is more important than adding unlimited API replicas. If write demand approaches the database ceiling, the next design work is data partitioning, asynchronous workflows, or a changed product consistency requirement, not another gateway.

15. Focused implementation example

Implement the algorithm after defining the policy. This Go example demonstrates a process-local token bucket for one protected route, not a global multi-instance quota.

type Bucket struct {
    mu       sync.Mutex
    tokens   float64
    capacity float64
    refill   float64
    last     time.Time
}

func (b *Bucket) Allow(cost float64, now time.Time) bool {
    b.mu.Lock()
    defer b.mu.Unlock()

    elapsed := now.Sub(b.last).Seconds()
    if elapsed < 0 {
        elapsed = 0
    }
    b.tokens = math.Min(b.capacity, b.tokens+elapsed*b.refill)
    b.last = now

    if cost <= 0 || b.tokens < cost {
        return false
    }
    b.tokens -= cost
    return true
}

Production code also needs bounded key cardinality, idle-key eviction, configuration validation, metrics, clear failure policy, and separate logic for invalid costs. For global limits, replace local state with atomic owned state or leased token batches. Do not put remote coordination under the mutex of a request-serving process.

16. Observability and operating signals

Observe demand, admission, placement, capacity, saturation, and membership together. A fleet average hides the endpoint that is about to fail.

Track distribution, not only totals: maximum-to-median endpoint load, coefficient of variation, per-zone utilization, and the fraction of traffic on the hottest 1% of endpoints. Use bounded labels. Tenant IDs can create dangerous metric cardinality, so retain sampled exemplars or aggregated heavy-hitter reports instead of an unbounded time series per tenant.

Actionable alerts

• Ready capacity cannot survive the configured failure domain.
• Desired capacity stays at the configured maximum while demand or queue age rises.
• Decision-to-ready time exceeds burst tolerance.
• One endpoint or zone carries materially more normalized work than peers.
• Discovery revision age or last-known-good use exceeds its safe bound.
• Rejection rate threatens the SLO, or one tenant consumes an abnormal share.
• Dependency concurrency reaches its ceiling while upstream continues admitting work.
• Drain deadlines expire and force cancellation or connection termination.

17. Security, privacy, and abuse resistance

Load balancers, discovery, gateways, and limiters cross trust boundaries and decide where valuable traffic and identity flow.

• Authenticate workloads that register endpoints. Authorize service identity, namespace, locality, and metadata updates separately.
• Encrypt control-plane and data-plane traffic where the threat model requires it. Rotate credentials without removing all endpoints together.
• Treat forwarding headers as untrusted unless a known proxy stripped and recreated them. Preserve verified identity in an integrity-protected channel.
• Validate host, path, header size, body size, decompressed size, method, and protocol upgrade before expensive routing or buffering.
• Apply limits before costly authentication where possible, then apply stronger identity-based limits after authentication.
• Avoid exposing internal topology, endpoint addresses, quota policy, or tenant activity in public errors and metrics.
• Protect the limiter state store from arbitrary high-cardinality keys. Attackers can create millions of identities or headers to exhaust memory.
• Audit policy changes and emergency bypasses. A hidden unlimited allow-list can defeat every capacity assumption.

18. Performance, capacity, and cost trade-offs

Optimize cost per successful SLO-compliant operation, not utilization alone. Driving every instance to 95% CPU may reduce machine count while dramatically increasing queueing, retries, and failure risk. Include load-balancer processing units, cross-zone bytes, control-plane calls, limiter storage, idle failure capacity, and rejected-work cost in the model.

19. Failure scenarios and troubleshooting

Evidence-first investigation

1. Compare offered, admitted, completed, failed, retried, and shed rates.
2. Find the first saturated resource in the request path, not the loudest downstream symptom.
3. Compare per-endpoint and per-zone work, latency, errors, readiness, and discovery revision.
4. Check whether capacity is ready, only requested, starting, draining, or stuck at a maximum.
5. Inspect connection age, protocol multiplexing, stickiness, and endpoint-set refresh.
6. Check dependency concurrency, queue age, pool wait, and retry amplification.
7. Mitigate by reducing low-priority load, enforcing admission, restoring warm capacity, then ramp.

20. Testing strategy

Test the controller, the data plane, and the failure transitions. A balanced steady-state chart does not prove safe scaling.

Use production-shaped request cost and connection lifetime. Closed-loop load generators wait for responses and can hide overload because their offered rate falls as latency rises. Use an open-loop generator when validating behavior at a fixed arrival rate, while keeping explicit safety boundaries in shared environments.

21. Hands-on exercises and design scenarios

Exercise 1: simulate the algorithms

Simulate 100 endpoints, including 10 half-capacity endpoints and requests whose durations follow a heavy-tailed distribution. Compare round robin, random, least requests, and power of two choices. Report p99 endpoint occupancy and maximum-to-median work. Expected reasoning: equal request counts fail under duration and capacity skew; normalized load improves placement; delayed metrics can destabilize strict least-load selection.

Exercise 2: build and test a distributed limiter

Implement a token bucket with one owner per tenant, then add leased token batches to ten gateway processes. Kill a gateway while it holds a lease and partition two gateways from the owner. Expected reasoning: define whether unused tokens are lost or reclaimable, bound overshoot, choose fail-open or fail-closed per operation, and measure latency versus exactness.

Exercise 3: plan a drain

Run an HTTP API with keep-alive plus WebSocket chat sessions. Remove 25% of instances. Expected reasoning: stop new selection first, signal long-lived clients to reconnect, checkpoint or externalize session state, enforce a maximum drain deadline, and measure forced disconnects.

Scenario: flash-sale inventory API

Demand rises 20 times in five seconds, writes are expensive, and overselling is unacceptable. A strong answer pre-provisions capacity, uses authenticated per-buyer and global admission, reserves concurrency for inventory, uses idempotency, rejects early rather than building a long queue, and separates browse reads from purchase writes. Autoscaling alone cannot meet the rise time, and adding stateless instances cannot exceed inventory storage capacity.

Scenario: multi-zone chat gateway

Millions of persistent connections make request-count balancing misleading. A strong answer balances new connections by normalized active connection cost, caps connections per gateway, externalizes durable message state, supports reconnect and resume, drains gradually, and keeps enough per-zone connection capacity for failure. It also handles a few unusually active rooms as hot keys rather than assuming every connection costs the same.

Scenario: public search API

Anonymous traffic and expensive queries create abuse risk. A strong answer combines network protection, per-IP-prefix coarse limits, authenticated tenant limits, weighted query cost, bounded execution time, result-size limits, query complexity guards, concurrency admission, and cheaper degraded results. It does not rely on IP as a fair user identity.

22. Interview questions and model answers

1. Vertical versus horizontal scaling?

Vertical scaling adds capacity to one resource and is operationally simple but bounded by machine size and a larger failure unit. Horizontal scaling adds independently schedulable resources and can improve capacity and fault isolation, but needs balancing, partitioning, and explicit state ownership. Most systems combine them. Follow-up: identify the shared bottleneck that stops linear scale-out.

2. Why are stateless API instances easier to scale?

Any ready instance can serve the next request, so instances can be added, removed, and replaced without moving correctness-critical local state. Durable state still exists in explicit stores. Follow-up: sessions, local files, schedulers, and long connections often reveal hidden state.

3. Scalability versus elasticity?

Scalability is the ability to handle growth by adding resources with acceptable efficiency. Elasticity is the time-dependent adjustment of resources to demand. A system may scale manually but have slow or unsafe elasticity. Follow-up: reaction time determines whether reactive scaling can catch a burst.

4. Why can autoscaling make an outage worse?

Scaling on latency from a saturated database can add callers and connections to that database. Noisy metrics and rapid scale-down can flap capacity. Cold new instances can fail readiness or receive too much traffic. Use demand-related signals, dependency concurrency ceilings, stabilization, warm-up, and failure-capacity bounds.

5. Layer 4 versus Layer 7 load balancing?

Layer 4 chooses a destination using transport flow information and supports broad protocols with low application visibility. Layer 7 understands an application protocol and can route individual requests using hosts, paths, or headers, but adds processing, trust, and protocol complexity. Follow-up: one multiplexed connection can make Layer 4 request distribution uneven.

6. When does round robin fail?

It assumes similar endpoint capacity and request cost. Long requests, heavy tenants, heterogeneous machines, sticky sessions, and multiplexed connections break that assumption. Least active work, weights, random sampling, or cost-aware routing may fit better.

7. What is power of two choices?

Randomly sample two ready endpoints and choose the less loaded after normalizing for capacity. It dramatically improves tail occupancy over one random choice without globally searching for the least-loaded endpoint. Follow-up: stale load signals and synchronized balancers still require smoothing and randomness.

8. Why use consistent hashing?

It keeps most key-to-endpoint mappings stable as membership changes, which helps cache locality and partition ownership. It does not solve endpoint health, hot keys, replication, or correctness. Define fallback and rebalancing behavior.

9. Liveness versus readiness?

Liveness asks whether restarting the process is appropriate. Readiness asks whether it should receive new traffic. Conflating them can restart every overloaded instance and amplify failure. Startup checks protect slow initialization before either decision becomes meaningful.

10. How do you remove an instance safely?

Mark it draining, stop new selection, wait for propagation, stop accepting new work, complete or transfer bounded in-flight work, close resources, then terminate. Account for DNS caches, connection reuse, streams, and a maximum deadline.

11. Registry versus DNS discovery?

Registries can provide watches, rich metadata, leases, and rapid updates but require a client or proxy integration. DNS is widely supported and cached but has multiple cache layers and limited real-time load information. Both can be stale, so data-plane health and last-known-good policy still matter.

12. Should discovery be queried for every request?

Usually no. It makes control-plane availability and latency part of every data-plane request. Clients or proxies should watch or poll asynchronously and use a bounded local endpoint set, exposing its revision and age.

13. What is wrong with sticky sessions?

They constrain balancing, concentrate heavy users, slow recovery, and tie availability to one endpoint. They can be a locality optimization, but correctness should survive loss of affinity. Externalize or reconstruct essential session state.

14. Explain token bucket.

Tokens refill at a configured rate up to a capacity. Work spends tokens. The refill rate controls long-run speed and capacity controls allowed burst. Weighted token cost represents unequal work. Follow-up: use atomic updates and a trusted monotonic time source.

15. Token bucket versus leaky bucket?

Token bucket permits a bounded burst after idle time. A queue-style leaky bucket releases at a steadier rate but adds wait and needs a bounded queue. Terminology varies, so state the exact algorithm and behavior rather than relying only on the name.

16. Fixed versus sliding windows?

Fixed counters are cheap but allow boundary bursts. Sliding logs are precise but store an event history. Sliding counters interpolate neighboring windows for lower cost and approximate rolling behavior. Choose based on the protected resource's burst tolerance.

17. How do you enforce a global distributed limit?

Options include a shared atomic counter, one owner per key, or token leases to local gateways. Exact centralized decisions cost latency and availability. Local leases improve performance but allow bounded approximation and stranded allowance. State the acceptable overshoot and failure policy.

18. Why is rate limiting not enough?

Ten requests per second can still create 1,000 in-flight operations if each lasts 100 seconds. Concurrency limits protect occupancy, while rate limits protect arrival speed and quotas protect longer allocations. Expensive dependencies commonly need all three.

19. Backpressure versus load shedding?

Backpressure asks upstream to slow or stop. Load shedding rejects selected work to preserve useful capacity. Backpressure works only if upstream honors it and queues remain bounded. At public boundaries, rejection is often the practical backpressure signal.

20. How much spare capacity is enough?

Start from the required failure model. For one-zone survival, verify that remaining zones support peak demand at tested safe per-instance capacity, then include balancing skew, rollout overlap, warm-up, recovery work, growth, and dependency limits. A fixed 20% rule without this model is not evidence.

21. Why reject requests during overload?

After saturation, accepting more work increases queues, latency, memory, timeouts, and retries, often reducing successful throughput. Early cheap rejection preserves capacity for useful work and lets clients back off. Follow-up: define priority and degraded modes before the incident.

22. How do you know balancing is good?

Compare normalized per-endpoint requests, bytes, active work, latency, errors, CPU, and zones. Use distributions such as maximum-to-median, not fleet averages. The goal is balanced constrained resource use and SLO-compliant completions, not identical request counts.

23. Concise cheat sheet

• Scaling supplies capacity; discovery publishes it; balancing places work; admission protects it.
• Scale the bottleneck, not merely the visible frontend.
• Stateless means replaceable compute, not an application without state.
• Shared-nothing reduces coordination but needs explicit partition ownership.
• Autoscaling is delayed feedback. Model reaction time and burst rise time.
• Keep minimum warm capacity and verify one-failure-domain capacity.
• Layer 4 usually places flows; Layer 7 can place application requests.
• Round robin assumes similar endpoint capacity and request cost.
• Power of two choices gives strong balance with cheap approximate information.
• Hashing provides affinity, not health or hot-key protection.
• Separate startup, liveness, readiness, draining, and slow-start behavior.
• Connection reuse improves efficiency but delays rebalancing and draining.
• Discovery views are always potentially stale. Keep revision, age, and last-known-good policy.
• DNS TTL is not an instant failover guarantee because caches and connections persist.
• Rate controls speed; quota controls allocation; concurrency controls occupancy.
• Token refill rate sets long-run speed; bucket capacity sets burst.
• Fixed windows have boundary bursts; sliding approaches reduce them at higher cost.
• Exact global limits require coordination. Leases trade exactness for speed and availability.
• Bound every queue. Reject work that cannot finish before its deadline.
• Protect dependencies with their own admission and concurrency limits.
• Return 429 for policy limiting and 503 for temporary service inability when useful.
• Measure offered, admitted, completed, retried, rejected, and shed work separately.
• Test spike, saturation, zone loss, stale discovery, cold recovery, and limiter failure.

24. Official references

• Kubernetes: Horizontal Pod Autoscaling
• Kubernetes: Liveness, Readiness, and Startup Probes
• Kubernetes: Service and Cloud-Native Service Discovery
• Kubernetes: DNS for Services and Pods
• Envoy: Load Balancing Algorithms
• Envoy: Slow Start Mode
• AWS Elastic Load Balancing: How Load Balancing Works
• AWS Elastic Load Balancing: Routing Algorithms and Target Behavior
• Google SRE Book: Load Balancing at the Frontend
• Google SRE Book: Load Balancing in the Datacenter
• Google SRE Book: Addressing Cascading Failures
• Amazon Builders' Library: Using Load Shedding to Avoid Overload
• NGINX: HTTP Request Limiting Module
• IETF RFC 6585: 429 Too Many Requests
• IETF RFC 2782: DNS SRV Records