Messaging, Event-Driven Architecture, and Stream Processing - Complete Notes

A language-neutral guide to messages, queues, event logs, delivery guarantees, consumer state, replay, schema evolution, stream-time reasoning, production operations, and interview design.

00. The mental model

Messaging moves durable facts or requested work through time. It lets a producer finish without requiring every consumer to be available at that exact moment.

01. The problem messaging solves

Direct RPC couples caller success to the immediate availability and latency of the callee. Messaging inserts a durable handoff. The producer can receive an acknowledgement after the broker stores the message, while consumers process later at a rate their capacity permits. This provides temporal decoupling, buffers finite bursts, supports fan-out, and preserves replayable evidence.

Messaging does not remove coupling. It changes its form:

• Producers and consumers still share event meaning, identifiers, and schema contracts.
• A queue moves waiting time into broker lag and may make failure less visible to a caller.
• Asynchronous completion requires status, reconciliation, and user-facing pending states.
• A durable broker becomes critical infrastructure with capacity and recovery obligations.
• Events can spread sensitive data to more stores and teams than a direct response would.

Choose the interaction from the business contract

02. Message and event contracts

A reliable event is self-identifying, traceable, versionable, and explicit about when and where the fact occurred.

{
  "id": "evt_01K...",
  "type": "com.shop.order.accepted.v2",
  "source": "urn:service:orders",
  "subject": "orders/ord_4821",
  "occurredAt": "2026-08-03T09:15:22.481Z",
  "publishedAt": "2026-08-03T09:15:22.509Z",
  "schemaVersion": 2,
  "correlationId": "checkout_701",
  "causationId": "cmd_988",
  "tenantId": "tenant_12",
  "contentType": "application/json",
  "data": {
    "orderId": "ord_4821",
    "customerId": "cus_91",
    "currency": "INR",
    "totalMinor": 249900
  }
}

• id gives one logical event a stable identity across redelivery.
• type describes business meaning, not the producing class name.
• source identifies the authority that emitted it.
• subject identifies the aggregate or entity and often guides routing.
• occurredAt is business event time; publishedAt is transport time.
• correlationId follows one business flow; causationId links cause.
• tenantId supports authorization, quotas, routing, and deletion workflows.
• schemaVersion is useful only with documented compatibility rules.

Do not place retry count, broker offset, or consumer-specific status in the immutable business payload. Those belong in transport headers or consumer state. Do not copy full customer records into every event when stable identifiers or purpose-limited fields are sufficient.

Commands and events are not interchangeable

ChargePayment has an intended handler, may be rejected, and should carry an idempotency identity. PaymentCharged says the charge already succeeded and must not be rejected by consumers. An event named ProcessPayment hides whether it is a fact or a request. An event should contain the data required to interpret the fact, but it should not expose an unstable internal object graph.

03. Broker internals and durable handoff

A producer serializes a message, selects a destination and often a partition key, then sends it to a broker leader. The broker validates authorization and size, appends bytes to storage, replicates according to policy, and acknowledges at a configured durability point. Consumers fetch or receive messages, apply business logic, make effects durable, and acknowledge or advance a position.

Producer       Broker leader       Replica        Consumer       Database
   | PUT m           |                |               |              |
   |---------------->| append         |               |              |
   |                 | replicate m    |               |              |
   |                 |--------------->| persist       |              |
   | ack accepted    |<---------------|               |              |
   |<----------------|                |               |              |
   |                 | deliver m      |-------------->| begin        |
   |                 |                |               | write effect |
   |                 |                |               |------------->|
   |                 |                |               | commit       |
   |                 | ack m          |<--------------|              |

Each arrow can fail independently. If the producer loses the broker acknowledgement, it may send the same logical message again. If the consumer commits its database change and crashes before acknowledgement, the broker redelivers. If the broker acknowledges before required replicas or storage are durable, an accepted message may disappear during failure. Correctness follows from explicitly handling every ambiguous boundary.

Push and pull delivery

• Pull lets consumers choose fetch rate, batch size, and position. It naturally exposes lag but may add polling latency when traffic is sparse.
• Push can reduce idle polling and simplify endpoints, but the broker needs delivery flow control, retry policy, endpoint authentication, and overload behavior.
• Either model needs bounded in-flight messages. Unlimited prefetch merely moves backlog into consumer memory where it is less durable and less observable.

04. Topics, partitions, keys, and ordering

A topic is a named logical stream. A partition is one independently ordered shard of that stream. Producers normally compute a partition from a key. All events for one order can use orderId, preserving order for that order while many orders process in parallel.

Ordering is usually guaranteed only within one partition and only as observed in that log. Retries can reorder sends unless the producer protocol sequences them. Concurrent consumers can finish effects out of order even when delivery was ordered. A slow message can block later work for the same partition. State the required scope precisely: per order, per account, per device, or global. Global order is expensive and rarely required.

Partition count is a capacity decision

More partitions allow more parallel consumers and write leaders, but increase metadata, connections, open files, replication work, recovery time, and cross-partition coordination. A consumer group can actively process no more partitions than it has useful members. Partition count should cover forecast peak throughput, failure headroom, and future consumers without creating tiny operational units. Changing the partition count may change key mapping, so designs that depend on stable partition placement need an explicit migration.

05. Consumers, groups, offsets, and rebalancing

A subscription identifies an independent copy of delivery. Within a competing consumer group, partitions or messages are assigned across members so work is divided. Different groups each observe the stream independently. An offset is a consumer's logical next position in a partition, not proof that every external side effect succeeded.

JOINING -> ASSIGNED -> FETCHING -> PROCESSING -> COMMITTING
   ^          |                         |              |
   |          +----- lease lost -------+--------------+
   +-------------------- REVOKED / REBALANCING -------+

Membership changes, crashes, timeouts, partition-count changes, and deployments can trigger a rebalance. Ownership transfers must fence the old owner before the new owner acts. During revocation a consumer should stop accepting new work, finish or cancel bounded in-flight work, persist safe positions, and release partition-local resources. Long blocking handlers can miss heartbeats and cause repeated rebalances, creating a feedback loop.

Commit position after the correct durable boundary

06. Delivery and processing guarantees

Delivery count, processing count, and business-effect count are different measurements.

At-most-once

A message is processed zero or one time. Loss is possible, duplication is avoided. This can be appropriate for replaceable telemetry or high-frequency presence signals where an older sample has little value.

At-least-once

A message is retried until acknowledged, so duplicates are possible. It is the common durable baseline because losing accepted business work is often worse than detecting a duplicate. It requires idempotent effects or explicit deduplication.

Effectively-once

Delivery may repeat, but the observable business result is the same as applying the logical message once. A unique constraint on (consumer, message_id) in the same transaction as the effect is a strong pattern. A remote email or card charge needs that remote system to honor a stable idempotency key, or it remains outside the atomic boundary.

Exactly-once claims

Ask: exactly once for which records, state, sink, failure model, and retention period? A stream engine may atomically checkpoint operator state and source positions. A broker may atomically publish output and commit input positions. Neither automatically includes an arbitrary database, HTTP API, email provider, or human action. Even a correct transaction can be observed more than once by a client that retries a response. Use the narrow product guarantee, then design idempotency and reconciliation at every boundary outside it.

BEGIN
  INSERT INTO consumed_message(consumer_name, message_id)
  VALUES ('inventory-projector', :message_id)
  ON CONFLICT DO NOTHING

  IF inserted_row_count = 1 THEN
    UPDATE inventory_view
       SET reserved = reserved + :quantity
     WHERE sku = :sku
  END IF
COMMIT

acknowledge message only after COMMIT succeeds

07. Idempotency, deduplication, and ambiguity

An operation is idempotent when repeating the same logical request has the same intended effect as performing it once. It does not mean every response byte or timestamp is identical. The identity must represent user intent, not a transport attempt. A retry must reuse the same key; a new intent needs a new key.

• Use a database uniqueness constraint, not a check-then-insert race.
• Store status and canonical result so duplicates receive the original outcome.
• Scope keys by tenant and operation to prevent cross-tenant collisions.
• Retain dedupe state at least as long as messages can be retried or replayed.
• Hash or validate request payloads so one key cannot silently represent two commands.
• Define what happens when processing is still in progress or previously failed.

A short in-memory cache is not durable deduplication. It is lost on restart and differs across replicas. A Bloom filter may cheaply reject probable repeats only if false positives cannot drop required work, or if it is followed by an authoritative lookup.

08. Retries, poison messages, and dead-letter handling

Retry only failures likely to change without modifying the message: temporary network failure, rate limiting, or dependency unavailability. Validation errors, unsupported versions, and broken invariants require correction, quarantine, or operator action. Use capped exponential backoff with jitter and a total attempt and age budget.

A dead-letter queue is a quarantine, not a trash can. Store original identity, destination, schema, failure classification, attempts, first and last failure time, sanitized diagnostics, and ownership. Protect it like production data. Provide tools to inspect, correct, redrive with the same identity, skip with approval, and audit every action. Alert on age and rate, not only count.

09. Retention, replay, and compaction

Queue messages are often removed after acknowledgement. A retained log keeps records by time or size regardless of individual readers; consumers store their own positions. Replay resets a position or writes into a controlled new destination. It is a production write workload, not a free read.

• Retention must exceed the longest expected outage, audit, and rebuild window.
• Replay capacity must not starve current traffic or overwhelm downstream side effects.
• Consumers need deterministic logic or versioned projections to explain changed results.
• External effects must remain idempotent during replay.
• Privacy deletion must account for logs, compacted values, backups, and derived stores.

Key compaction retains a recent value per key but is not immediate and does not necessarily erase every historical byte at once. Tombstones express deletion in a compacted stream and must remain long enough for slow consumers to observe them. Compaction is useful for rebuilding current configuration or entity state; time retention is needed for a complete audit history.

10. Schema evolution and compatibility

Events outlive deployments. Producers and consumers roll independently, retained records may be replayed years later, and several schema versions coexist. A schema registry can store versions, identifiers, and compatibility policy, but governance still requires ownership and semantic review.

Renaming a field is usually remove plus add. Changing rupees to paise without changing the field name is syntactically compatible but semantically catastrophic. Changing an enum from open text to a closed generated type can break older readers. For breaking meaning, publish a new event type or version, dual-publish during migration if necessary, observe consumers, backfill safely, and retire only after retention and rollback windows close.

11. Stream-processing model

A stream processor continuously reads records, transforms them, groups by key, maintains state, and emits results. Stateless map and filter operations are easy to replay. Stateful joins, aggregations, and pattern detection require durable state plus a consistent input position.

order events
  -> validate envelope
  -> assign event timestamp
  -> key by merchantId
  -> five-minute tumbling window
  -> aggregate accepted value and failure count
  -> join merchant risk tier
  -> emit dashboard metric and anomaly event

Event time, processing time, and ingestion time

• Event time is when the business event occurred at the source.
• Ingestion time is when platform infrastructure first accepted the event.
• Processing time is the worker clock when an operator handles it.

Processing-time windows are simple and low latency, but results change with queueing, restart, and replay. Event-time windows better match business questions such as sales per minute, but require timestamps, out-of-order policy, watermarks, state retention, and late-data correction.

Windows

Watermarks and late data

A watermark is an estimate that event time has progressed to a value. When the watermark passes a window end, the engine may emit its result. It is not proof that no older event can arrive. Choose allowed lateness from measured source delay and business correction cost. Late events can be dropped with a metric, routed to a repair stream, or update a previously emitted result through an upsert or retraction. Idle input partitions must be identified, or one silent partition can hold the combined watermark back indefinitely.

arrival order:  event(10:00:05), event(10:00:02), watermark(10:00:04)

The 10:00:02 event is valid before the watermark arrives.
An event for 10:00:03 arriving after this watermark is late by policy.
The system must define drop, side output, or correction behavior.

12. Stateful recovery and checkpoints

Stateful processing must recover operator state and input positions to one consistent logical point. A checkpoint captures keyed state, timers, source positions, and sometimes in-flight data. On failure, workers restore a completed checkpoint and replay later records. Replay is expected; the framework's guarantee depends on coordinated source, state, and sink behavior.

• Checkpoint interval trades recovery replay against steady storage and coordination cost.
• Checkpoint timeout detects snapshots that cannot finish, but repeated failure needs action.
• Backpressure can delay aligned barriers and make recovery protection stale.
• Savepoints support planned migration but do not replace regular failure checkpoints.
• State schema evolution must be tested before deploying code that restores old snapshots.
• External sinks need idempotent upsert, transactions, or a commit protocol.

If a job counts each input in restored operator state exactly once, that does not mean each record physically traversed the network once. Recovery may replay it, while the restored state and source position make the final result equivalent to one application. Be precise in interviews and design documents.

13. Batching, flow control, and backpressure

Batching amortizes syscalls, compression, replication, and network overhead. Larger batches improve throughput until they increase wait time, memory, retry size, and tail latency too far. Batch by both count and bytes, then flush on a time bound. A ten-message batch of images and a ten-message batch of IDs do not cost the same.

Backpressure is a feedback signal that slows upstream production. A bounded consumer should limit fetched bytes, in-flight records, processing concurrency, and per-key queues. If an upstream source cannot slow, the system needs admission control, durable spill with a strict limit, or load shedding. Pausing fetch does not fix a dependency that remains permanently below arrival rate.

semaphore = asyncio.Semaphore(40)

async def handle(message):
    async with semaphore:
        event = validate(message)
        result = await provider.send(
            key=event.id,
            recipient=event.data["recipient"],
            deadline=event.data["expiresAt"],
        )
        await store_outcome(event.id, result)
        await message.ack()

# The adapter must stop fetching when the bounded task set is full.
# ack happens only after the durable outcome.

14. Complete order-event pipeline

Assumptions: 2,000 orders per second at peak, bursts to 5,000, strict order lifecycle per order, no duplicate charge, analytics may be eventually consistent, and accepted orders must survive a zone failure.

1. The order API validates authentication, price version, and an idempotency key. In one local transaction it writes the order and an outbox record.
2. An outbox relay publishes OrderAccepted keyed by order ID. Duplicate publication is allowed because event ID is stable.
3. The replicated broker acknowledges only after the required durable replica policy. Partitions distribute order keys across zones.
4. Payment, inventory, notifications, search, and analytics use independent consumer groups. One slow analytics deployment cannot consume payment's position.
5. Payment uses event ID as a scoped idempotency key with the payment provider and stores canonical outcome before acknowledgement.
6. Inventory atomically records message ID and reservation. It emits a new outcome through its own outbox rather than publishing inside an open database transaction.
7. A workflow correlates outcomes and transitions order state. Timeouts create explicit events and late outcomes enter reconciliation rather than silently overwriting terminal state.
8. Analytics uses event time and an allowed-lateness policy. Corrections upsert by window and key.

Failure decisions

15. Notification worker design

Notification commands use a work queue because one worker should attempt each channel delivery. Partition by a hash that spreads recipients, but use a recipient-specific sequence when channel order matters. The payload carries template ID and safe variables, not rendered secrets. A worker validates consent, current preferences, expiry, and tenant authorization before calling the provider.

async function deliver(job: NotificationJob): Promise<void> {
  const prior = await outcomes.find(job.messageId);
  if (prior?.terminal) return;

  const policy = await preferences.authorize(job.tenantId, job.recipient, job.channel);
  if (!policy.allowed || Date.now() >= job.expiresAt) {
    await outcomes.recordTerminal(job.messageId, "suppressed");
    return;
  }

  const result = await provider.send(job, { idempotencyKey: job.messageId });
  await outcomes.recordTerminal(job.messageId, result.providerId);
}

Retriable provider failures go to delayed retry with jitter. Invalid addresses and permanent rejection become terminal outcomes. Tenant quotas prevent one campaign from delaying security notifications. Metrics separate accepted, delivered, provider-rejected, suppressed, expired, retrying, and unknown. Delivery receipts are events, not proof a human read the message.

16. Observability and troubleshooting

Queue depth alone is insufficient. A quiet queue may be healthy or may have stopped accepting writes. A large retained log is normal; consumer lag is the relevant signal. Monitor each stage from production through useful business effect.

• Produce request rate, bytes, batch size, compression, errors, retries, and acknowledgement latency.
• Broker disk, network, replica health, under-replication, leader changes, and partition skew.
• Consumer lag in records, bytes, and estimated time, plus oldest unprocessed event age.
• Fetch rate, processing latency, acknowledgement latency, rebalance count, and assignment churn.
• Retry and dead-letter rate by reason, attempt, event type, version, tenant, and deployment.
• Stream watermark delay, late records, checkpoint duration, failure, state size, and backpressure.
• Useful outcomes such as paid orders and sent notifications, not only technical acknowledgements.

Trace context may cross asynchronous boundaries, but one event can fan out to thousands of spans. Use links from consumer spans to producer context, sample intentionally, and preserve message ID, partition, offset, consumer group, and schema version in structured logs. Do not place full message payloads, tokens, or personal data in labels or logs.

Lag investigation playbook

1. Confirm whether arrival rate rose or service rate fell and identify affected partitions.
2. Compare oldest age with business deadlines; record count alone hides message cost.
3. Check broker throttling, replica health, consumer rebalances, pauses, and fetch limits.
4. Inspect handler latency, downstream saturation, hot keys, poison retries, and recent changes.
5. Stop retry amplification and nonessential producers before scaling blindly.
6. Add consumers only when partitions and dependencies have capacity.
7. Recover in controlled ramps; verify useful throughput and reconcile expired work.

17. Security, privacy, and trust boundaries

• Authenticate producers, consumers, brokers, administrators, and replay tools separately.
• Authorize publish and consume per destination, consumer identity, environment, and tenant scope.
• Encrypt transport and storage; rotate keys without making retained data unreadable unexpectedly.
• Validate size, type, schema, required claims, and safe numeric ranges before allocation or use.
• Treat every message as untrusted input even when it came from an internal service.
• Protect against decompression bombs, parser abuse, injection, deserialization gadgets, and SSRF.
• Use quotas for bytes, records, partitions, connections, retries, and replay to resist abuse.
• Do not trust a payload tenant ID without binding it to authenticated producer authority.
• Minimize personal data and document retention, residency, access, deletion, and legal hold.
• Audit destination changes, permission changes, offset resets, DLQ redrives, and data exports.

Event fan-out expands the trust boundary. A consumer should receive only fields needed for its purpose. Encryption at the broker does not prevent an authorized but overprivileged consumer from reading data. For highly sensitive fields, use tokenization, purpose-specific topics, field-level encryption with controlled keys, or fetch-on-demand through an authorization boundary.

18. Capacity, scalability, latency, and cost

Estimate records per second, average and p99 record bytes, replication factor, retention, read fan-out, compression, peak-to-average ratio, and failure headroom. One 2 KB event at 20,000 events per second is about 40 MB/s before protocol overhead. With replication factor three, broker-side network and disk write work is at least roughly 120 MB/s before indexes and filesystem effects. Seven-day raw retention is about 24.2 TB before compression:

ingress_bytes_per_second = events_per_second * average_event_bytes
replicated_write_rate = ingress_bytes_per_second * replication_factor
retained_bytes = ingress_bytes_per_second * retention_seconds / compression_ratio
consumer_read_rate = ingress_bytes_per_second * number_of_full-stream_groups
catch_up_time = backlog_bytes / spare_consumer_bytes_per_second
required_service_rate >= peak_arrival_rate / target_utilization

Add protocol overhead, index and segment metadata, compaction work, uneven partitions, page cache, retries, and re-replication. Size for losing a broker or zone while maintaining the acknowledgement policy. Network egress from many consumer groups can cost more than ingress. Long retention makes replay and audit powerful but increases storage, compliance scope, recovery time, and deletion complexity.

19. Failure scenarios and safer corrections

20. Testing strategy

Unit and property tests

• Envelope and schema validation, including unknown fields, bounds, and malicious payloads.
• Idempotency state transitions for new, in-progress, completed, failed, and expired keys.
• Partition-key stability and distribution using realistic skewed identifiers.
• Window assignment, watermark, late-data, deduplication, and timer behavior.
• Property: any number of identical deliveries creates at most one terminal business effect.

Integration and contract tests

• Use a real compatible broker and database, not only mocks of acknowledgement behavior.
• Verify old and new producer and consumer versions against retained contract fixtures.
• Crash after receive, after effect commit, before acknowledgement, and during rebalance.
• Test relay duplicates, broker failover, revoked ownership, and expired retention.
• Restore stream state from the previous deployed checkpoint format.

Load, fault, and recovery tests

• Measure throughput and p99 under representative message-size and key skew distributions.
• Burst above capacity and verify bounded memory, honest lag, and protected dependencies.
• Lose a broker or zone and verify acknowledgement durability and reduced-capacity headroom.
• Pause one downstream dependency and confirm retries do not create a storm.
• Build a retention-sized backlog, catch up safely, and keep current traffic healthy.
• Replay a full projection and compare checksums, counts, and business invariants.

21. Design decision checklist

1. Define command, event, request-response, or continuous-query semantics.
2. Name the source of truth and atomic transaction boundary.
3. Choose work sharing, fan-out, or retained replay behavior.
4. Define message identity, aggregate key, ordering scope, and partition strategy.
5. State durability acknowledgement, replication, retention, and disaster assumptions.
6. Choose delivery guarantee and business-effect deduplication.
7. Specify acknowledgement, retry, poison-message, deadline, and redrive behavior.
8. Define schema ownership, compatibility, rollout, and retirement.
9. Bound batch, bytes, in-flight work, lag, and downstream concurrency.
10. Plan observability, security, privacy, capacity, replay, and recovery tests.

22. Hands-on exercises

Exercise 1: duplicate-safe inventory consumer

Design tables and pseudocode for inventory reservations. Inject a crash after the inventory update but before broker acknowledgement. Expected reasoning: message identity and effect must commit atomically; redelivery detects the prior result; acknowledgement follows commit; insufficient stock is a stored terminal outcome, not a transient retry.

Exercise 2: partition a chat stream

Require message order within a conversation, 100,000 concurrent rooms, and one celebrity room at 20% of traffic. Expected reasoning: conversation ID gives correct normal ordering but creates a hot key. Discuss a dedicated partition, ordered substreams with sequence merge, admission control, or relaxing order for independent event types. Do not silently hash one conversation across shards.

Exercise 3: repair late analytics

Process mobile purchase events that can arrive 30 minutes late. Expected reasoning: use event time, observed delay distribution, watermark policy, allowed lateness, upsertable window results, late side output, and a batch reconciliation path. Compare latency and state cost at 2, 10, and 60 minutes of allowed lateness.

Exercise 4: disaster replay

Rebuild a search projection from seven days of events without hurting checkout. Expected reasoning: isolated consumer identity, rate and concurrency limits, replay capacity, deterministic versioned transformation, shadow destination, completeness checks, cutover, rollback, and privacy deletions applied before exposure.

23. Interview questions and model answers

1. Queue versus publish-subscribe versus log?

A work queue divides tasks among competing workers. Publish-subscribe gives independent subscriptions their own delivery. A retained log stores ordered partitions and lets consumers own positions and replay. Products can support several modes, so describe required semantics rather than selecting by label.

2. Does a message broker make a system loosely coupled?

It reduces temporal and location coupling, but producers and consumers remain coupled through semantics, schema, identifiers, ordering, and operational expectations. Undocumented events can create hidden coupling worse than a versioned API.

3. What does at-least-once mean?

The system retries unacknowledged messages, so an accepted message should eventually be delivered under stated failure assumptions, but delivery and processing can repeat. Consumers must make effects idempotent or deduplicate durably.

4. How do you prevent duplicate payments?

Carry one stable payment-intent identity through every retry, use it as a scoped idempotency key at the payment authority, store canonical outcomes, and query ambiguous results. Broker offset alone cannot prevent duplicates at an external provider.

5. Why commit the offset after the database transaction?

Committing first can lose the effect if the process crashes. Committing the effect first can cause redelivery, so the database transaction must also record message identity or use an idempotent operation. Then redelivery is safe.

6. What ordering does a partitioned log provide?

It normally provides one append order within each partition. There is no single order across partitions. Consumer concurrency, retry, and external side effects can still complete out of order, so key by the smallest entity that needs order and serialize effects for that key.

7. What happens during a consumer-group rebalance?

Assignments are revoked and redistributed as membership changes. Consumers must stop old ownership, settle bounded work, store safe positions, and resume from committed positions. Slow handlers and missed heartbeats can cause churn; cooperative transfer can reduce disruption but does not remove fencing needs.

8. How do you handle a poison message?

Classify it, stop unbounded immediate retry, preserve evidence, and quarantine it with ownership. Decide whether order allows later records to continue. Fix data or code, then redrive using the same identity under rate limits and audit.

9. What is event time?

It is when the business event occurred, unlike processing time when a worker handles it. Event time gives stable replayable analytics but requires watermarks, late-event policy, and retained state.

10. What is a watermark?

It is a progress estimate for event time. When it passes a window, the engine can emit results. Older events can still arrive, so the design must define allowed lateness and correction. A slow or idle input can hold the combined watermark back.

11. Can exactly-once processing call an HTTP API exactly once?

Not automatically. Framework transactions usually cover their source, state, and supported sink. An arbitrary HTTP side effect is outside that boundary. It needs a stable idempotency key, outcome query, durable intent, and reconciliation.

12. How do you choose a partition key?

Start from ordering and state-locality requirements, then test distribution against real skew. Prefer the narrow business entity that needs ordering. Plan for hot keys, tenant isolation, resharding, and the fact that changing partition count may remap keys.

13. What does consumer lag tell you?

It estimates unprocessed distance between produced and consumed positions. Track records, bytes, and oldest age because costs vary. Lag reveals imbalance but not root cause; correlate arrival, processing, dependency, broker, and rebalance signals.

14. How do schemas evolve safely?

Define backward, forward, and semantic compatibility; use contract checks; prefer additive optional fields; roll readers before writers when required; support mixed versions; and give breaking meaning a new version or event type. Retained old events must remain readable.

15. Why is a DLQ not enough?

It prevents one bad message from retrying forever, but it can silently lose business work. A safe quarantine needs alerts, ownership, diagnostics, retention, correction, audited redrive, completeness checks, and a decision about ordering.

16. Design an order event pipeline.

Clarify throughput, ordering, durability, duplicate, and latency requirements. Use an atomic outbox, replicated partitioned log keyed by order, independent consumer groups, durable idempotency at effects, versioned envelopes, bounded retries, quarantine, lag objectives, reconciliation, secure ACLs, and recovery tests. State every exact transaction boundary.

Follow-up: What changes if analytics can lose data?

Analytics might use a cheaper at-most-once path or shorter retention, but payment and inventory guarantees should remain independent. The answer must follow explicit business loss tolerance, not a broker-wide default.

24. Revision cheat sheet

• A command requests an action; an event states an immutable past fact.
• Queues divide work, subscriptions fan out, and retained logs support independent replay.
• Messaging reduces temporal coupling but adds schema and operational coupling.
• Broker acknowledgement point determines accepted-message durability.
• Ordering is normally per partition, not global and not automatically per external effect.
• Key by the smallest entity that requires ordering and test real skew.
• Consumer offsets are positions, not proof of business completion.
• Commit effects before acknowledgement and make redelivery safe.
• At-least-once permits duplicates; effectively-once deduplicates observable effects.
• Ask exactly once within which transaction domain and failure model.
• Stable idempotency identity must survive retries and replays.
• Bound retries by attempts, age, backoff, jitter, and business deadline.
• A DLQ is an owned, monitored, auditable quarantine.
• Retention enables outage recovery and replay but increases cost and privacy scope.
• Schema compatibility includes meaning, not only parseability.
• Event time supports replayable business results; processing time follows runtime delay.
• A watermark is a progress estimate and needs a late-data policy.
• Stateful recovery coordinates state, timers, source position, and sink behavior.
• Batch by count and bytes with a time bound.
• Backpressure must propagate; otherwise shed or durably bound work.
• Observe oldest event age, useful outcomes, retries, rebalances, and checkpoint health.
• Replay is a production workload that needs isolation and correctness checks.
• Use atomic outbox for database-to-broker dual writes.
• Secure every producer, consumer, administrative reset, and redrive path.

25. Primary official references

• Apache Kafka official documentation - topics, partitions, replication, producers, consumers, delivery, retention, and compaction.
• Apache Kafka protocol guide - requests, responses, record batches, offsets, and group coordination protocol details.
• CloudEvents core specification - interoperable event context attributes, identity, source, type, subject, time, and data.
• AsyncAPI specification - language-neutral documentation for channels, operations, messages, servers, and security.
• Apache Flink official time documentation - event time, processing time, watermarks, and late-event reasoning.
• Apache Flink official checkpointing documentation - state, source replay, durable checkpoint storage, and processing guarantees.
• Reactive Streams specification - asynchronous stream processing with non-blocking backpressure.
• Java SE Flow API - publisher, subscriber, subscription, demand, and cancellation interfaces.
• Python standard library asyncio queues - bounded asynchronous queues, waiting producers, task completion, and shutdown behavior.
• Protocol Buffers language guide - field evolution, unknown fields, reserved identifiers, enums, and compatibility constraints.
• IETF RFC 9110: HTTP Semantics - idempotent methods and the limits of request retry assumptions at HTTP boundaries.