Networking, DNS, TCP, TLS, HTTP, REST, and RPC - Complete Notes

A language-neutral guide to how a secure request moves from a name in a client to a backend response, including protocol internals, failure behavior, API choices, production operations, and interview reasoning.

00. The end-to-end mental model

A network request is a sequence of agreements between layers, not one indivisible action.

An application starts with a logical destination such as https://api.example.com/orders/42. DNS maps the host name to one or more addresses. IP moves packets across networks. TCP or QUIC creates transport behavior between endpoints. TLS authenticates a peer and protects bytes. HTTP gives those protected bytes request and response meaning. REST or RPC then defines the application contract. A load balancer, proxy, firewall, NAT device, or service mesh may participate at several points.

Application:  GET /orders/42 plus headers
HTTP:         request semantics and message framing
TLS:          encrypted records with integrity protection
TCP:          ordered byte stream split into segments
IP:           packets addressed between hosts
Link:         frames delivered over each local hop

On receive, each layer removes and validates its own envelope.
Routers normally inspect IP, not the HTTP order identifier.

01. Network foundations

Internet Protocol offers best-effort packet delivery. Reliability, security, and application meaning are added by other layers.

Packets, routers, and hops

A sender does not usually know the complete physical path. It selects a next hop using a routing table. Each router examines the destination IP address, chooses another next hop, and decrements the IPv4 TTL or IPv6 Hop Limit. The packet may be dropped, duplicated, delayed, or reordered. IP itself does not promise delivery or ordering.

Routing control protocols build reachability information, while the data plane forwards actual packets. A default route handles destinations without a more specific match. Longest-prefix match selects the most specific matching route. Internet routes can change while a connection is alive, so packets from one conversation need not take identical paths.

Sockets, addresses, ports, and connection identity

A socket is an operating-system communication endpoint exposed to a process. A server commonly creates a socket, binds it to a local address and port, listens, and accepts connections. A client creates a socket and connects to a remote address and port. A TCP connection is distinguished by protocol plus source address, source port, destination address, and destination port.

server listens on 203.0.113.10:443

client A 192.0.2.20:53001 -> 203.0.113.10:443
client B 192.0.2.20:53002 -> 203.0.113.10:443
client C 198.51.100.7:41000 -> 203.0.113.10:443

The server port can be the same because each connection tuple is different.

Client source ports are usually temporary, or ephemeral. They are finite. High connection churn, long TIME_WAIT retention, a small NAT port pool, or too many destinations behind one translated address can exhaust them. File descriptor limits, accept queues, connection tracking, memory, and application worker limits can be exhausted before network bandwidth is full.

MTU, fragmentation, and path MTU

A link maximum transmission unit, or MTU, limits the largest IP packet carried without fragmentation on that link. The smallest supported size on an end-to-end path is the path MTU. IPv4 routers may fragment when permitted. IPv6 routers do not fragment packets in transit, so an IPv6 source uses fragmentation when necessary. Modern transports try to avoid IP fragmentation because losing one fragment prevents reassembly of the whole original packet and middleboxes may mishandle fragments.

Path MTU Discovery depends on receiving control messages that a packet is too large. Blocking all ICMP can create a black hole: small packets work, while large requests or TLS records repeatedly disappear. Datagram protocols also need application-level size discipline. For TCP, maximum segment size negotiation and segmentation normally hide packet boundaries from the application.

NAT, firewalls, and proxies

Network address translation, or NAT

Rewrites addresses and often ports while keeping temporary mapping state. It conserves public IPv4 addresses and creates an addressing boundary, but it is not the same as authorization.

Stateful firewall

Applies policy using addresses, ports, protocol, direction, and observed connection state. A timeout can discard idle state even while endpoints believe a connection is open.

Forward proxy

Acts for clients, commonly controlling or observing outbound access.

Reverse proxy

Acts in front of servers, commonly terminating TLS, routing requests, enforcing limits, caching, and hiding backend topology.

Tunnel

Carries bytes without interpreting the inner application protocol, although its setup may use another protocol such as HTTP CONNECT.

Every stateful middlebox has capacity and timeouts. A keepalive interval longer than a NAT idle timeout does not keep the mapping alive. A proxy may buffer a streaming response, change connection reuse, normalize headers, or apply a smaller payload limit than the origin. Draw each hop explicitly during diagnosis.

02. DNS resolution, caching, and operations

DNS is a delegated, distributed, cached database that maps names to typed resource records.

Stub, recursive, root, TLD, and authoritative roles

An application normally calls a local stub resolver. The stub asks a recursive resolver, which may answer from cache. On a miss, the recursive resolver follows referrals, typically from a root server to a top-level-domain server and then to an authoritative server for the zone. The authoritative server answers from zone data. Iterative resolution means each server points the resolver toward the next authority. Recursive service means the resolver performs that work for the client.

client stub       recursive        root         .example TLD       authority
    | query A         |              |                |                 |
    |---------------->|              |                |                 |
    |                 | query        |                |                 |
    |                 |------------->|                |                 |
    |                 | referral to TLD               |                 |
    |                 |<-------------|                |                 |
    |                 | query                         |                 |
    |                 |------------------------------>|                 |
    |                 | referral to authority                           |
    |                 |<------------------------------|                 |
    |                 | query A                                          |
    |                 |------------------------------------------------>|
    |                 | A 203.0.113.10, TTL 60                           |
    |                 |<------------------------------------------------|
    | answer + TTL     |                                                   |
    |<----------------|                                                   |

Later clients may receive the cached answer until its remaining TTL expires.

Common resource records

TTL, caches, propagation, and negative answers

Each resource record carries a TTL that tells caches how long the data may be reused. An update is visible immediately at the updated authority, but resolvers holding an older answer can keep it until their cached TTL expires. This behavior is often called propagation, although no global push occurs. Lower the TTL before a planned migration, wait for the old TTL population to age out, make the change, and keep the old destination serving long enough for stragglers.

Negative answers such as a proven nonexistent name can also be cached. Creating a record shortly after clients received NXDOMAIN may therefore remain invisible to them temporarily. TTL zero does not guarantee that every client immediately re-queries, because applications, operating systems, libraries, proxies, and resolvers may add independent caching behavior.

DNS traffic distribution

Returning several addresses, weighted answers, geography-aware answers, or anycasted service addresses can influence destination selection. DNS does not observe each final connection, and recursive resolver caching can aggregate many clients behind one answer. A removed address may still receive traffic until caches expire. DNS distribution therefore needs healthy overlapping capacity and an application-level failover plan.

Common DNS failures and diagnosis

• NXDOMAIN: the queried name does not exist, possibly due to a typo or stale negative cache.
• SERVFAIL: resolution failed, commonly because of unreachable authority, validation failure, or delegation trouble.
• Timeout: packets, responses, or TCP fallback were blocked or servers were overloaded.
• Lame delegation: parent delegation points to a server that is not authoritative.
• Missing glue: resolver cannot reach an in-bailiwick authoritative name without address records from the parent.
• Truncation failure: a large UDP response requests TCP retry, but TCP port 53 is blocked.
• Split-horizon mismatch: internal and external clients intentionally receive different data, but query the wrong resolver.
• DNSSEC validation failure: signatures, keys, delegation records, or time validity do not form a valid chain.

1. Record the exact name, type, resolver, result code, answer, and TTL.
2. Query the configured recursive resolver.
3. Query a second known resolver to separate local cache from authority.
4. Trace referrals from root to authority.
5. Query each authoritative server directly.
6. Compare serials, records, DNSSEC status, UDP and TCP behavior.
7. Check application and operating-system caches.
8. Correlate the answer with connection attempts and destination health.

03. TCP state, reliability, and limits

TCP turns an unreliable packet path into a reliable, ordered, full-duplex byte stream between two endpoints.

Three-way handshake and state

client                                      server
CLOSED                                      LISTEN
  |                                             |
  | SYN, sequence x                             |
  |-------------------------------------------->| SYN-RECEIVED
  | SYN + ACK, sequence y, acknowledge x + 1    |
  |<--------------------------------------------|
  | ACK, acknowledge y + 1                      |
  |-------------------------------------------->| ESTABLISHED
ESTABLISHED                                    |

Sequence numbers identify byte positions. SYN consumes sequence space.

The handshake proves two-way reachability, establishes initial sequence numbers, and negotiates options. It costs a network round trip before normal application bytes, unless a higher-level mechanism safely combines early data. A server protects its pending connection queue from exhaustion through sizing, admission control, SYN cookies where appropriate, and upstream filtering.

Acknowledgements, retransmission, ordering, and ambiguity

TCP numbers bytes and acknowledges received sequence space. A sender retains unacknowledged data and retransmits after loss is inferred through timers or acknowledgement patterns. The receiver reorders segments and presents only an ordered stream to the application. Checksums detect common corruption, but TLS or application integrity is still required against attackers.

An acknowledgement means bytes reached the peer TCP stack, not that the peer application committed a payment. A client timeout after sending a request is ambiguous: the server may not have seen it, may be processing it, may have committed it, or may have sent a response that was lost. Application idempotency keys and status lookup solve this ambiguity. Transport reliability alone cannot.

Flow control versus congestion control

The sender can have at most the smaller of the receive window and congestion window in flight, subject to implementation details. A connection begins cautiously and increases sending as the path demonstrates capacity. Long-distance high-bandwidth paths need enough window to fill their bandwidth-delay product. A tiny application read buffer or stalled consumer can advertise a zero window even when the network is healthy.

Close, reset, half-close, and TIME_WAIT

TCP directions close independently with FIN. A graceful close allows already-sent bytes to be consumed. A reset, or RST, aborts the connection and can discard unread data. The active closer commonly enters TIME_WAIT so delayed old segments cannot be confused with a later connection using the same tuple and so the final acknowledgement can be retransmitted.

A successful write followed by a later reset still does not reveal whether the remote application acted. Do not map FIN, RST, timeout, or cancellation directly to a business outcome. Record the request identity and make the operation safely repeatable or queryable.

Keepalive and connection pooling

TCP keepalive probes detect a dead peer after a configurable idle interval. They are different from HTTP persistent connections and application heartbeats. Defaults are often too slow for request deadlines and may be longer than middlebox idle timeouts.

A connection pool amortizes DNS, transport, and TLS setup, but must be bounded per destination and globally. It needs connect, acquisition, idle, lifetime, and request deadlines. Pools should retire connections before known infrastructure lifetime limits, validate stale connections, and expose pending waiters. A pool with 100 connections does not imply 100 useful concurrent requests: HTTP/2 and HTTP/3 multiplex streams, while a downstream database might permit only 20 operations.

04. UDP and QUIC trade-offs

UDP provides datagrams with minimal transport behavior. QUIC builds secure, reliable, stream-oriented transport in user space over UDP.

UDP

UDP preserves datagram boundaries and adds ports and a checksum to IP delivery. It does not establish a connection, retransmit, order datagrams, pace a sender, or provide application backpressure. That simplicity fits DNS queries, telemetry where fresh samples replace old ones, and applications that implement specialized real-time recovery. A responsible UDP application still needs congestion control, amplification defenses, size limits, authentication when needed, and explicit loss and reordering behavior.

QUIC

QUIC version 1 integrates TLS 1.3, reliable streams, stream and connection flow control, loss recovery, and congestion control over UDP. Connection IDs allow a connection to survive some address changes, such as a mobile client changing networks, after path validation. Most QUIC protocol control information is encrypted, reducing interference from middleboxes and allowing implementations to evolve in user space.

QUIC removes cross-stream transport head-of-line blocking: loss affecting one stream does not stop delivery on an unrelated stream. Ordered bytes within the affected stream still wait, and connection-level congestion control still shares path capacity. HTTP/3 clients need a TCP-based fallback where UDP connectivity fails.

05. TLS identity, encryption, and termination

TLS protects a connection only after the client validates the intended peer and both sides derive fresh traffic keys.

TLS 1.3 handshake

client                                                    server
  | ClientHello: versions, key share, SNI, ALPN              |
  |--------------------------------------------------------->|
  | ServerHello: selected version and key share               |
  | {EncryptedExtensions, Certificate, CertificateVerify,     |
  |  Finished}                                                |
  |<---------------------------------------------------------|
  | verify chain, hostname, signature, validity, policy        |
  | {Finished}                                                |
  |--------------------------------------------------------->|
  | encrypted application data                                |
  |<========================================================>|

Braces indicate messages protected with handshake keys.

Ephemeral key agreement establishes a shared secret without sending the traffic keys. The certificate binds a public key to names under a trust model. CertificateVerify proves possession of the corresponding private key and binds the handshake transcript. Finished authenticates the transcript under derived secrets. ALPN selects an application protocol such as HTTP/2, and SNI tells a multi-tenant endpoint which server name the client seeks.

Certificates, trust chains, and hostname verification

A client validates a path from the presented end-entity certificate through intermediate certificate authorities to a configured trust anchor. It verifies signatures, validity times, constraints, key usages, and local policy. It must separately match the reference host name from the URI against permitted identifiers in the certificate. A cryptographically valid chain for the wrong host is not sufficient.

• Send the required intermediate certificates, but normally not the client's root trust anchor.
• Monitor expiry from the real serving endpoint, not only a certificate inventory.
• Automate renewal and test reload because issuance success does not mean serving success.
• Keep private keys in a narrowly authorized boundary and rotate after suspected compromise.
• Do not disable verification to repair a missing intermediate or name mismatch.
• Account for clock error because certificate validity and some revocation data are time-bound.

Session resumption and early data

TLS 1.3 resumption uses a pre-shared key derived from an earlier authenticated connection, reducing handshake work and latency. Session tickets require key lifecycle and rotation across terminating instances. A resumption key exposed across a large fleet expands the impact of compromise.

Zero round-trip-time early data can be replayed by an attacker even though its confidentiality is protected. Accept it only for operations safe under replay, and use protocol and application controls. A payment mutation or one-time token exchange should not rely on early-data delivery as proof of uniqueness.

Mutual TLS

With mTLS, both endpoints present credentials and prove key possession. It can establish workload identity between services, but does not by itself decide authorization. Map the authenticated identity to explicit policy, rotate client credentials, propagate end-user identity separately, and log the identity used for a decision. Avoid treating every certificate from a broad internal authority as permission to perform every operation.

TLS termination patterns

06. HTTP semantics

HTTP defines reusable semantics for requests and responses independently of whether messages use HTTP/1.1 text framing, HTTP/2 frames, or HTTP/3 over QUIC.

Request target, method, fields, content, and response

POST /orders HTTP/1.1
Host: api.shop.example
Content-Type: application/json
Accept: application/json
Idempotency-Key: 8c44...
Content-Length: 58

{"sku":"A-17","quantity":2,"shippingAddressId":"addr-9"}

HTTP/1.1 201 Created
Location: /orders/ord-318
Content-Type: application/json
ETag: "order-v1"
Cache-Control: no-store

{"id":"ord-318","status":"accepted"}

A field's meaning comes from its specification, not its spelling. Intermediaries may process and transform messages subject to HTTP rules. End-to-end fields describe the selected representation or resource semantics. Hop-by-hop behavior applies to one connection and must not be blindly forwarded.

Methods, safety, and idempotency

Safe means the client did not request a state-changing effect. Logging and accounting may still occur. Idempotent means multiple identical intended requests have the same intended effect as one, not that every response is byte-identical. A DELETE retry may return 404 after the first request succeeded and remain idempotent.

Business idempotency is explicit. For order creation, store an idempotency key with a normalized request fingerprint and final outcome in the same correctness boundary as order creation. A repeated key with the same request returns the saved outcome. The same key with different content is rejected. Expiry must exceed the realistic retry and reconciliation window.

Status codes by outcome

High-value headers and metadata

• Host or :authority selects an origin on shared infrastructure.
• Content-Type describes sent content; Accept expresses acceptable response types.
• Content-Length declares content size where used; conflicting framing is dangerous.
• Authorization carries credentials and must not be logged in plaintext.
• Cache-Control, validators, and Vary govern reusable responses.
• Location identifies a redirect target or newly created resource, depending on status.
• Retry-After can guide retry timing for specific statuses, but clients still need bounds and jitter.
• Trace context and correlation fields support diagnosis, but are not proof of caller identity.

07. HTTP caching and conditional requests

HTTP caching is a protocol-level reuse contract between origin, private caches, and shared caches.

Freshness determines whether a stored response can be reused without contacting the origin. Validation asks the origin whether a stored response still represents the resource. A cache key starts with the target URI and method, then may vary using selected request fields named by Vary. Forgetting an authorization-relevant or representation-selecting dimension can leak one user's content to another.

Cache-Control: max-age=60

The response is fresh for 60 seconds relative to its calculated age.

Cache-Control: private

A private cache may store it; a shared cache should not.

Cache-Control: no-store

Caches must not store the request or response under the normal HTTP caching rules.

ETag

An opaque validator for the selected representation.

Last-Modified

A time validator, generally less precise than a strong entity tag.

client/cache                              origin
  | GET /catalog/A-17                       |
  | If-None-Match: "catalog-v8"             |
  |---------------------------------------->|
  | 304 Not Modified                        |
  | ETag: "catalog-v8"                      |
  |<----------------------------------------|
  | reuse stored representation             |

304 transfers metadata, not the full representation.

Conditional mutation prevents lost updates. A client reads ETag: "order-v4", edits, then sends If-Match: "order-v4". If another writer created version 5, the server rejects the stale mutation instead of silently overwriting it.

08. HTTP/1.1, HTTP/2, and HTTP/3

The versions share HTTP semantics but encode and transport messages differently.

HTTP/1.1 requires unambiguous message framing. Conflicting Content-Length/Transfer-Encoding interpretations across proxy and origin can enable request smuggling. Normalize strictly, reject ambiguity, keep components patched, and test multi-hop parsing.

HTTP/2 multiplexes many streams but needs limits for concurrent streams, header list size, frame rate, and flow-control state. A single busy connection can create uneven load when a proxy chooses a backend once per connection. Graceful shutdown uses protocol signaling so new streams move elsewhere while accepted streams finish.

HTTP/3 maps one request-response exchange to a QUIC stream and uses QPACK for field compression. QUIC connection and stream flow-control limits must be tuned. UDP reachability, CPU cost, load balancer support, connection ID routing, and observability tooling belong in the rollout plan.

09. REST, JSON, Protobuf, and RPC

REST and RPC are contract styles. JSON and Protocol Buffers are representation formats. HTTP is a protocol. Choose each dimension independently from requirements.

REST constraints and resource modeling

REST is an architectural style defined by constraints including client-server separation, stateless requests, cacheability, a uniform interface, a layered system, and optional downloaded code. A RESTful HTTP API models addressable resources and uses standard HTTP semantics rather than treating every request as an unrelated tunnel.

POST   /orders                 create order
GET    /orders/ord-318         retrieve current order representation
PUT    /orders/ord-318/address replace the shipping-address subresource
DELETE /orders/ord-318/hold    release an existing hold
GET    /orders?cursor=...      traverse a collection

An exceptional domain action can still be modeled explicitly:
POST /orders/ord-318/cancellation-requests

Stateless means each request carries the information needed to understand it. It does not mean the server stores no data. Pagination, filtering, error representation, idempotency, authorization, and concurrency behavior are part of the contract. Hypermedia can advertise state-dependent next actions, though many production APIs use documented URI conventions instead.

JSON versus Protocol Buffers

In Protobuf, field numbers are the wire identity. Never reuse a removed number or name; reserve them. Adding fields is generally compatible because receivers can ignore unknown fields, but correctness still depends on semantics, default behavior, enum evolution, presence, and validation. A schema-compatible change can still break a business invariant.

RPC and gRPC

RPC presents communication as invoking an operation on another process. The local-call appearance is useful but incomplete: remote calls have latency, partial failure, serialization, deadlines, authentication, and uncertain outcomes. Interfaces should expose these realities.

syntax = "proto3";

service Inventory {
  rpc Reserve(ReserveRequest) returns (ReserveReply);
  rpc WatchAvailability(WatchRequest) returns (stream Availability);
}

message ReserveRequest {
  string request_id = 1;
  string sku = 2;
  int32 quantity = 3;
  reserved 4;
}

message ReserveReply {
  string reservation_id = 1;
  string status = 2;
}

gRPC commonly uses HTTP/2 transport and Protocol Buffers contracts. It supports unary, server-streaming, client-streaming, and bidirectional-streaming methods. Metadata carries call-level information. Deadlines bound usefulness, cancellation signals abandoned work, and status codes communicate RPC outcomes. Cancellation is cooperative and does not roll back a completed side effect.

10. Streaming, WebSocket, and Server-Sent Events

Real-time delivery is a lifecycle and backpressure problem, not only a choice of wire protocol.

HTTP response streaming

Sends a bounded or unbounded response incrementally. Useful for large exports and streamed records. Proxies must not buffer unexpectedly.

gRPC streaming

Provides typed one-way or bidirectional message streams with transport flow control. The application still needs bounded queues and slow-consumer policy.

WebSocket

Begins with an HTTP handshake and becomes a full-duplex framed connection. Good for interactive chat or collaborative state where both sides send at arbitrary times.

Server-Sent Events, or SSE

Streams UTF-8 events from server to browser using text/event-stream. Built-in event IDs and reconnection fit one-way notifications.

For every long-lived stream define authentication refresh, reconnect backoff, resume position, duplicate handling, ordering scope, heartbeat behavior, maximum message size, idle timeout, per-tenant connection quota, slow-consumer limits, deploy draining, and what happens when history needed for resume has expired.

server sends: id=1041, event=order-status, data=packed event
connection breaks after client receives but before it renders
client reconnects with last processed id=1040
server replays 1041 and later
client applies event only if event id is newer than stored processed position

Transport reconnection can duplicate delivery. Application state makes replay safe.

11. Deadlines, cancellation, retries, and limits

A timeout limits one wait. A deadline states when the entire result stops being useful.

Propagate the remaining deadline across service hops, reserving time for response transfer and cleanup. Each hop must stop queued and in-flight work when cancellation is observed. If a server ignores cancellation, abandoned work continues consuming capacity and can amplify an outage.

client total budget:                         800 ms
edge processing and network:                  80 ms
orders service queue and work:               180 ms
inventory call allocated from remaining:     250 ms
database work:                               120 ms
response and safety reserve:                 170 ms

At every hop:
remaining = original_deadline - monotonic_elapsed
if remaining < minimum useful work, reject before starting.

Retry only when the failure is plausibly transient, the operation is safe to repeat, the remaining deadline permits it, and a bounded retry budget protects the dependency. Use exponential backoff with jitter. A transport error before receiving a response does not prove the server did nothing. For mutations, combine an idempotency key with persisted outcome and reconciliation.

Payload and metadata limits

Limit request line or pseudo-header size, total header bytes, field count, compressed and decompressed content, individual message size, stream duration, and expansion ratio. Apply compatible limits at CDN, load balancer, gateway, framework, parser, and business layer. Reject oversized bodies before buffering them into memory, stop reading when a streaming limit is reached, and return a stable error without reflecting sensitive content.

12. Complete secure request trace

Trace every boundary to understand latency, identity, connection reuse, and ambiguous failure.

Assume a mobile client sends GET https://api.shop.example/orders/ord-318. The edge load balancer terminates public TLS, authenticates a token, and uses mTLS to an orders backend.

1. The client parses the URI into scheme, host, default port 443, path, and query. It checks local policy, proxy configuration, existing connections, and DNS caches.
2. The stub resolver asks its recursive resolver for A and AAAA records. The recursive resolver returns a cached answer or follows delegations to the authoritative zone. Each answer has a remaining TTL.
3. The client chooses a reachable address. Its operating system selects a source address and ephemeral port, uses a route to a next hop, and resolves local link addressing as needed.
4. A stateful firewall and NAT may create connection state. For TCP, client and edge complete the three-way handshake. For HTTP/3, QUIC performs transport and cryptographic setup over UDP.
5. The TLS ClientHello names api.shop.example using SNI and offers ALPN protocols. The edge selects a protocol, presents a certificate chain, proves private-key possession, and both sides derive traffic keys.
6. The client validates the certificate chain, validity, hostname, signatures, constraints, and policy. Only then does it treat the connection as authoritative for the HTTPS origin.
7. The client sends the HTTP request on a persistent connection or new multiplexed stream with an authorization credential, accepted representation, trace context, and deadline.
8. The edge enforces parser, header, payload, abuse, and authentication policy. It removes or overwrites untrusted forwarding identity fields instead of blindly trusting client input.
9. The load balancer selects a healthy orders instance. It obtains or reuses a backend connection, validates the backend certificate for the expected workload identity, and forwards a reduced deadline and authenticated caller context.
10. The orders service authorizes access to ord-318, reads state, constructs the representation, and returns status, fields, validator, and cache policy.
11. Each proxy records its own timings and forwards the response. TLS protects each configured hop. TCP or QUIC performs retransmission and flow control. The client validates message framing and content type, then presents the result.
12. Connections remain reusable until lifetime, idle, health, certificate, deploy-drain, or protocol policy closes them. Request completion does not require connection closure.

DNS lookup or cache decision
+ pool acquisition
+ network connect
+ TLS or QUIC handshake
+ request upload
+ edge queue and policy
+ backend pool acquisition and network
+ backend queue and application work
+ response time to first byte
+ response transfer
= end-to-end latency

On reused connections, connect and handshake can be zero for that request.
Their cost still appears during churn, deploys, expiry, and failure.

13. Production architecture and operations

Operate the request as a chain of independently bounded, observable hops.

Capacity and connection budgets

peak arrival rate = 8,000 requests/second
mean time in service = 0.12 seconds
mean in-flight requests = 8,000 * 0.12 = 960

If dependency latency rises to 2 seconds:
in-flight requests = 8,000 * 2 = 16,000

Without bounded admission, memory, streams, workers, and connections grow
before useful throughput improves.

Budget accepted connections, active streams, file descriptors, ephemeral ports, NAT mappings, TLS handshakes per second, connection memory, receive and send buffers, proxy queues, and backend pool size. Keep failure headroom. A zone loss sends more connections and handshakes to surviving zones exactly when their caches and pools may be cold.

Deployment and draining

• Fail readiness before stopping so no new work is assigned.
• Stop accepting new connections or streams while allowing bounded in-flight work to finish.
• Use protocol-specific graceful signals such as connection close policy or GOAWAY where applicable.
• Keep the grace period within caller deadlines and orchestration termination limits.
• Cancel remaining work safely, close listeners, and preserve idempotency for retried mutations.
• Stagger restarts to avoid simultaneous DNS, TLS, and connection storms.

Layered observability

Use route templates such as /orders/{id}, not raw identifiers, as metric labels. Redact credentials, cookies, query secrets, personal data, certificate subject details when sensitive, and payloads by default. Correlation identifiers connect hops but must be validated for size and syntax and must not grant trust.

Troubleshooting by first failing layer

Can the exact name and record type resolve?
  no  - inspect resolver, delegation, authority, TTL, DNSSEC
  yes - record every returned address

Can transport connect to an address and port?
  no  - inspect route, firewall, NAT, listener, backlog, packet loss
  yes - measure connect latency and selected address family

Can TLS authenticate and negotiate?
  no  - inspect time, SNI, chain, hostname, trust, ALPN, versions
  yes - record certificate fingerprint and negotiated protocol

Does HTTP/RPC produce a valid response?
  no  - inspect framing, proxy, timeout, reset, limits, status
  yes - inspect authorization and domain correctness

Is the result consistently correct?
  no  - compare destination instance, cache, version, tenant, retry history

14. Security, privacy, and abuse

Every resolver, proxy, parser, credential boundary, and backend connection is a trust boundary.

• Name resolution: use DNSSEC where its authenticity guarantee is required, secure registrar and DNS control-plane accounts, restrict zone transfers, and monitor unauthorized changes.
• TLS: require supported protocol versions, validate names and chains, automate rotation, protect keys, and do not use encryption as a substitute for authorization.
• Proxy identity: strip client-supplied forwarding and identity headers at the first trusted boundary, then add authenticated internal context.
• Parser differentials: reject ambiguous HTTP framing, duplicate security fields, invalid field syntax, and inconsistent URL normalization across gateway and application.
• SSRF: validate allowed destinations after parsing and resolution, block private and metadata networks as policy requires, control redirects, and defend against DNS rebinding.
• Amplification: validate reachability before large UDP responses, rate-limit unauthenticated work, and avoid response sizes far larger than requests.
• Resource abuse: bound headers, bodies, decompression, streams, connections, handshake CPU, request rates, concurrent work, and idle time.
• Privacy: minimize IP, hostname, query, SNI, certificate, and payload logging. Encrypt sensitive hops and define retention, access, and deletion.

15. Performance, scalability, and cost

Optimize round trips, reusable state, bytes, and queueing only after measuring the end-to-end critical path.

A rough transfer lower bound is payload bytes divided by available throughput, but latency also includes propagation, handshakes, congestion windows, queueing, serialization, and application time. Small requests are often round-trip bound. Large transfers are often throughput and congestion bound. Tail latency worsens under packet loss and saturation.

Count egress charges, cross-zone or cross-region traffic, load balancer processing, public address use, certificate and key operations, DNS queries, logging volume, and idle connection memory. Compression can trade compute cost for egress savings. A chat system with millions of mostly idle connections needs connection capacity and heartbeat efficiency even at low message throughput.

16. Failure scenarios and safer corrections

17. Testing strategy

Test contracts, wire behavior, capacity limits, failures, and recovery at the boundaries where production can disagree.

Unit and contract tests

• URI parsing, canonicalization, status mapping, content negotiation, and cache policy.
• JSON and Protobuf validation, unknown fields, missing presence, enum evolution, and reserved fields.
• Idempotency-key fingerprinting, replayed outcomes, key collision, expiry, and mismatched payload.
• Deadline arithmetic using a monotonic clock and cancellation propagation.
• Certificate name and identity mapping policy with valid and invalid fixtures.

Integration and interoperability tests

• Resolve against authoritative and recursive test DNS, including NXDOMAIN and TTL expiry.
• Exercise IPv4, IPv6, TCP, QUIC fallback, proxies, NAT-like idle expiry, and path MTU constraints.
• Test TLS with missing intermediate, wrong name, expired certificate, rotated trust, ALPN mismatch, and mTLS denial.
• Run HTTP/1.1, HTTP/2, and HTTP/3 clients against the real proxy chain and reject ambiguous framing.
• Verify REST and gRPC compatibility with older and newer generated clients.
• Test WebSocket, SSE, and RPC streams through production-equivalent proxy buffering and timeouts.

Load, fault, and recovery tests

• Measure cold and warm DNS, full and resumed TLS, new and reused connections separately.
• Find sustainable requests, streams, connections, handshakes, bytes, and messages per second.
• Inject delay, loss, reordering, reset, half-close, blackholed packets, and blocked UDP.
• Exhaust pool, file descriptor, ephemeral port, NAT mapping, stream, header, and body limits safely.
• Rotate certificates and DNS answers during traffic, then drain and restart every hop.
• Cancel clients after request upload and confirm abandoned backend work stops or reconciles.
• Soak long-lived connections through key rotation, deploys, network change, and idle periods.
• Verify dashboards and alerts identify the first failing layer rather than only reporting 5xx.

18. Focused implementation examples

Language APIs expose protocol controls, but the correctness principle remains independent of the library.

Go: one bounded HTTP client

var client = &http.Client{
    Timeout: 800 * time.Millisecond,
}

func fetchOrder(parent context.Context, id string) (*http.Response, error) {
    ctx, cancel := context.WithTimeout(parent, 750*time.Millisecond)
    defer cancel()

    url := "https://api.shop.example/orders/" + url.PathEscape(id)
    req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
    if err != nil {
        return nil, err
    }
    req.Header.Set("Accept", "application/json")
    return client.Do(req)
}

Reusing the Go client allows its transport to pool connections. Context cancellation bounds the request lifecycle. Production code still needs response body closure, size-limited decoding, stable error handling, TLS policy, observability, and carefully scoped retry behavior.

Node.js: consume a response with a byte limit

async function readLimited(response, maximumBytes) {
  const reader = response.body.getReader();
  const chunks = [];
  let received = 0;

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    received += value.byteLength;
    if (received > maximumBytes) {
      await reader.cancel("response too large");
      throw new Error("response exceeds configured limit");
    }
    chunks.push(value);
  }
  return Buffer.concat(chunks.map((chunk) => Buffer.from(chunk)));
}

A declared Content-Length can support early rejection but must not be the only control. Streaming and compressed content can exceed expectations, so enforce the actual decoded and encoded limits required by the trust model.

Java: explicit socket phases

try (Socket socket = new Socket()) {
    socket.connect(
        new InetSocketAddress("inventory.internal.example", 7443),
        300);
    socket.setSoTimeout(500);

    // Application framing and TLS are still required for a real protocol.
    // A successful connect proves only that the transport endpoint accepted.
}

A socket read timeout bounds blocked reads, not the complete operation. Higher-level code should carry one end-to-end deadline, secure the channel, define message framing, and cancel useful work when that deadline expires.

19. Design scenario: order API and live status

Choose protocol behavior from workload assumptions instead of declaring one API style universally best.

Assumptions

• Public mobile and web clients create and read orders using HTTPS.
• Peak traffic is 12,000 reads and 2,000 creates per second across three zones.
• Order creation must not duplicate an order after an ambiguous retry.
• Public p99 target is 400 ms for reads and 800 ms for accepted creation.
• Clients may receive order-status updates within five seconds, with reconnect support.
• Internal inventory calls need typed contracts and a 250 ms deadline.

First-pass decisions

1. Publish a resource-oriented JSON API for broad client compatibility. Use POST with an idempotency key for creation, GET with ETag for reads, and stable problem details for errors.
2. Return 201 when the order and initial state are durably created. If processing becomes asynchronous, return 202 with a status resource and state that acceptance is not completion.
3. Use SSE for one-way status delivery because browsers need server-to-client updates but do not send interactive messages on the same channel. Carry event IDs for resume and deduplicate.
4. Use unary gRPC for the controlled inventory boundary, with Protobuf evolution rules, mTLS workload identity, explicit authorization, propagated deadline, and request ID.
5. Terminate public TLS at a multi-zone edge and re-encrypt to backends. Strip untrusted forwarding headers. Limit requests and streams per caller before costly authentication and decoding.
6. Keep bounded per-destination connection pools. Size them from measured concurrency and surviving zone capacity, not from traffic averages.

Failure review

20. Hands-on exercises

1. Trace one HTTPS request with a cold DNS cache and no connection. Count network round trips for DNS, TCP, TLS 1.3, and the first response byte under a simplified one-RTT-per-stage assumption. Repeat with cached DNS and a reusable connection.
2. Design a DNS migration from address A to B when the current TTL is one hour. Include lowering TTL, waiting, overlap, rollback, monitoring, and final cleanup.
3. A proxy has 20,000 active HTTP/2 streams on 200 backend connections. Explain why connection count alone misses load and propose stream, pool-wait, and destination metrics.
4. Model a payment POST whose response is lost. Write pseudocode for an idempotency record, fingerprint check, atomic business effect, saved outcome, retry, and reconciliation.
5. Compare REST plus JSON, gRPC plus Protobuf, WebSocket, SSE, and asynchronous messaging for order creation, inventory lookup, chat, live notifications, and analytics ingestion.
6. Create a certificate failure lab with wrong hostname, expired leaf, missing intermediate, untrusted root, invalid client identity, and ALPN mismatch. Predict each observable failure.
7. Inject a path MTU black hole. Explain why a handshake or small request might work while a large response stalls, then list transport and packet evidence.
8. Calculate average in-flight requests at 5,000 RPS and 80 ms. Recalculate at 1.2 seconds and design bounded admission for the overload case.
9. Design a resume protocol for a notification stream. State ordering scope, event retention, duplicate handling, slow-consumer policy, and behavior when the cursor is too old.
10. Threat-model an image-fetch URL API for SSRF, DNS rebinding, redirect escape, large content, decompression, private-address access, and credential forwarding.

21. Interview questions and model answers

1. What happens after entering an HTTPS URL?

Parse the URI, check caches and policy, resolve the name, choose an address and route, establish TCP or QUIC, negotiate TLS and validate identity, send HTTP, traverse proxies, authorize and process at the backend, then return the response. Mention connection reuse because cached DNS and an existing secure connection can skip setup.

Follow-up: Which stages can be cached or reused, and which trust checks remain?

2. Recursive versus authoritative DNS?

A recursive resolver performs resolution for clients and caches results. An authoritative server serves original data for a zone. A resolver may answer from cache without contacting the authority. An authoritative server does not generally walk the global tree for the client.

Follow-up: Why can an updated record remain unseen until TTL expiry?

3. Why is DNS not a precise request load balancer?

Answers are cached by recursive resolvers and applications, many clients can share one cache, clients choose among addresses differently, and DNS does not observe every request or live connection. Use it for coarse distribution and combine it with healthy capacity and Layer 4 or Layer 7 balancing.

Follow-up: How would you safely remove one destination?

4. What does TCP guarantee?

TCP gives an ordered, reliable byte stream with duplicate suppression, retransmission, flow control, and congestion control while the connection operates. It does not preserve application message boundaries, encrypt content, or prove that acknowledged bytes produced a business commit.

Follow-up: Why can a client timeout leave a payment outcome unknown?

5. Flow control versus congestion control?

Flow control prevents a sender from overrunning the receiver's buffer. Congestion control limits traffic to what the network path can sustain. Application backpressure is a third layer that protects workers and dependencies even when transport buffers have space.

Follow-up: What signals show each kind of pressure?

6. What is head-of-line blocking?

Ordered delivery means later data waits for missing earlier data. HTTP/1.1 pipelined responses wait in response order. HTTP/2 multiplexes frames, but all streams share TCP ordered delivery, so packet loss can stall them. QUIC provides independent ordered streams, so loss normally blocks only affected stream data.

Follow-up: Does QUIC eliminate every kind of head-of-line blocking?

7. What does a TLS client verify?

It validates a certificate path to a trusted anchor, signatures, time validity, constraints, key use, local policy, and that an allowed certificate identifier matches the intended host or service identity. It also verifies handshake proofs. Encryption without peer verification is vulnerable to interception.

Follow-up: Why is an unexpired certificate still possibly invalid?

8. Is mTLS authorization?

No. mTLS authenticates the peer identity under a certificate trust model and protects the channel. Authorization separately decides whether that identity can perform an action on a resource. Broad certificate trust without narrow policy can create excessive privilege.

Follow-up: How do you rotate client identities without downtime?

9. Safe versus idempotent HTTP methods?

Safe methods request read-only semantics. Idempotent methods have the same intended effect when repeated as when performed once. GET is both. PUT and DELETE are idempotent but not safe. POST is neither by default, though an application can make a POST operation replay-safe with an idempotency contract.

Follow-up: Can an idempotent retry return a different response?

10. Freshness versus validation?

A fresh cached response can be reused without contacting the origin. A stale stored response can be conditionally validated with ETag or Last-Modified. A 304 response confirms reuse while avoiding the full representation transfer.

Follow-up: How can a wrong Vary policy leak tenant data?

11. REST versus RPC?

REST applies architectural constraints and a uniform resource interface. RPC exposes named remote operations. REST over HTTP often fits public, cache-aware interoperability. Typed RPC often fits controlled internal contracts and streaming. Neither removes the need for deadlines, compatibility, authentication, idempotency, and failure semantics.

Follow-up: Which would you use for browser clients and internal inventory, and why?

12. How do Protobuf schemas evolve safely?

Add fields with new numbers, tolerate unknown fields, preserve the semantics of existing fields, and reserve removed numbers and names. Review presence, defaults, enum evolution, validation, and old-client behavior. Wire compatibility alone does not guarantee business compatibility.

Follow-up: Why must a removed field number never be reused?

13. Timeout versus deadline?

A timeout bounds one duration, such as connect or read wait. A deadline is an absolute end for the complete operation and should propagate as remaining budget. Without propagation, each service can spend the full timeout and continue work after the caller has abandoned it.

Follow-up: Why should downstream timeouts be smaller than remaining client budget?

14. WebSocket versus SSE?

WebSocket is full-duplex framed communication and fits interactive two-way traffic. SSE is a server-to-client event stream with browser reconnection behavior and event IDs, fitting notifications. The real design also depends on proxies, authentication refresh, resume, slow consumers, limits, and expected connection scale.

Follow-up: How do you deploy without dropping every stream at once?

15. How can connection pooling fail?

Pools can exhaust, queue callers invisibly, retain stale or imbalanced connections, exceed NAT or server capacity, and create reconnect storms. Bound them, set acquisition and lifecycle limits, align with downstream capacity, drain gracefully, and measure active, idle, pending, age, and destination distribution.

Follow-up: How does HTTP/2 change pool sizing?

16. What does a 504 prove?

It proves a gateway did not receive a timely upstream response under its policy. It does not prove the upstream performed no side effect. For a mutation, the business outcome may be unknown and requires an idempotency key, operation status, or reconciliation.

Follow-up: When is an automatic retry safe?

17. Why might small requests work but large ones hang?

A path MTU problem can drop packets above a threshold while smaller packets pass. If required ICMP feedback is blocked, the sender may not adapt. Also check proxy body limits, flow control, decompression, and application buffering because similar symptoms can occur above IP.

Follow-up: What packet-level evidence separates MTU trouble from application limits?

18. How do you debug network latency?

Decompose DNS, pool wait, connect, TLS, upload, proxy queue, backend queue, application work, time-to-first-byte, and transfer. Split by address family, destination, protocol, reuse, status, and region. Correlate retransmissions, flow-control stalls, saturation, and business outcome.

Follow-up: Why is average end-to-end latency insufficient?

19. Where should TLS terminate?

Choose from the trust and operation model. Edge termination centralizes routing and security controls. Re-encryption protects backend hops. Passthrough preserves the backend as the public cryptographic endpoint but limits Layer 7 inspection. State who can see plaintext and how every backend identity is verified.

Follow-up: What changes when regulated data crosses the edge-to-backend hop?

20. Does HTTP/3 always beat HTTP/2?

No. HTTP/3 can reduce setup latency, avoid cross-stream TCP loss blocking, and support migration, but performance depends on loss, round trips, CPU, implementation, UDP reachability, connection reuse, and workload. Measure production-shaped clients and retain fallback.

Follow-up: Which metrics would justify or stop an HTTP/3 rollout?

22. Concise cheat sheet

• DNS names, IP addresses, and authenticated service identities are different.
• IP is best effort; transports and applications add stronger behavior.
• A TCP connection uses protocol plus source and destination addresses and ports.
• TCP is an ordered byte stream, not a message protocol.
• Flow control protects the receiver; congestion control protects the network path.
• Transport acknowledgement does not prove business commit.
• Pool acquisition, ports, descriptors, NAT state, and streams are finite resources.
• DNS TTL controls cache reuse, so changes need overlapping destinations and time.
• Authoritative servers own zone data; recursive resolvers find and cache answers.
• QUIC integrates TLS 1.3 and independent streams over UDP.
• TLS needs chain and hostname or service-identity verification.
• mTLS authenticates a peer; authorization still decides permission.
• HTTP semantics remain consistent across HTTP/1.1, HTTP/2, and HTTP/3.
• Safe describes requested semantics; idempotent describes repeated intended effect.
• ETag supports cache validation and conditional mutation.
• REST is an architectural style; RPC is an operation-oriented interaction style.
• JSON is readable; Protobuf is compact and schema-driven.
• Reserve removed Protobuf field numbers and preserve business semantics.
• WebSocket is full duplex; SSE is a server-to-client event stream.
• Propagate a deadline, cooperate with cancellation, and bound retries.
• Limit headers, payloads, decoded size, streams, connections, and work.
• A timeout during mutation creates an unknown outcome, not proof of failure.
• Measure every network phase and diagnose from the first failing layer.

23. Official references and standards

• RFC 8200: Internet Protocol, Version 6 Specification
• RFC 9293: Transmission Control Protocol
• RFC 5681: TCP Congestion Control
• RFC 768: User Datagram Protocol
• RFC 9000: QUIC, A UDP-Based Multiplexed and Secure Transport
• RFC 9002: QUIC Loss Detection and Congestion Control
• RFC 1034: Domain Names - Concepts and Facilities
• RFC 1035: Domain Names - Implementation and Specification
• RFC 2308: Negative Caching of DNS Queries
• RFC 7766: DNS Transport over TCP
• RFC 4033: DNS Security Introduction and Requirements
• RFC 8446: The Transport Layer Security Protocol Version 1.3
• RFC 5280: Internet X.509 Public Key Infrastructure Certificate Profile
• RFC 9525: Service Identity in TLS
• RFC 9110: HTTP Semantics
• RFC 9111: HTTP Caching
• RFC 9112: HTTP/1.1
• RFC 9113: HTTP/2
• RFC 9114: HTTP/3
• RFC 6455: The WebSocket Protocol
• WHATWG HTML: Server-Sent Events
• Roy Fielding: Representational State Transfer architectural style
• Protocol Buffers: Language Guide
• Protocol Buffers: Encoding
• gRPC: Core concepts, architecture, and lifecycle
• gRPC: Deadlines
• Java SE 21: Socket API
• Go standard library: net/http
• Node.js: HTTP API