Distributed Transactions, Sagas, Outbox, and Reliable Workflows - Complete Notes

A language-neutral guide to preserving business invariants across independently failing services, choosing between atomic commit and semantic recovery, building sagas and durable workflows, publishing database changes reliably, and operating the complete design in production.

00. Mental model and precise terminology

A distributed business operation is not one large database transaction. It is a conversation among owners of separate durable state, where every participant can succeed, fail, pause, or return an ambiguous answer independently.

The core task is to preserve a stated business invariant despite partial failure. For an order, the invariant might be: a confirmed order has one successful payment authorization and one valid inventory reservation; every non-confirmed order eventually releases any reservation and voids or refunds any payment it no longer needs. The word eventually is important. Intermediate states are often visible, so the API, user experience, accounting, and operations model must represent them honestly.

01. Atomicity boundaries and the dual-write problem

A local commit cannot atomically include an unrelated database, broker, HTTP service, email provider, or payment network unless all resources participate in one supported commit protocol.

What local ACID does and does not promise

Inside one database transaction, atomicity makes a group of writes visible together or not at all. Consistency means valid application and database rules move data between valid states. Isolation constrains interference from concurrent transactions. Durability preserves a committed result according to the database's durability policy. These properties end at that resource's transaction boundary. An HTTP call made from inside a database transaction is not automatically rolled back when the database rolls back.

BEGIN database transaction
  INSERT order(id = O-701, status = 'CREATED')
COMMIT

PUBLISH OrderCreated(O-701)

Crash after COMMIT, before PUBLISH:
  order exists, event is missing

If PUBLISH happens first and the database commit then fails:
  event exists, order is missing

Reversing the two operations only reverses the failure window.

The same problem appears when a service updates two independent databases, charges a card and creates an order, writes an object and metadata, or acknowledges a message before its database commit. A process-level mutex, synchronized block, or try/catch cannot make two durable systems commit atomically. A database callback that runs after commit also has a crash window.

A timeout produces knowledge, not rollback

Order service              Payment service               Payment database
     | authorize K-91 ---------->|                              |
     |                            | insert authorization K-91 -->|
     |                            |<------------- commit success |
     |                            | send APPROVED                 |
     |       connection lost X---|                              |
     | timeout: UNKNOWN           |                              |

UNKNOWN is not DECLINED. The safe next action is query(K-91) or retry(K-91),
where K-91 identifies the same authorization attempt.

Three broad consistency tools

02. Two-phase commit from first principles

Two-phase commit, or 2PC, separates a participant's durable promise from the coordinator's final decision so all prepared participants can reach the same commit or abort outcome.

Roles and durable records

• The application begins one global transaction with a globally unique transaction ID.
• The transaction manager or coordinator tracks enlisted resource managers.
• Each resource manager executes local work on behalf of the global transaction.
• Coordinator and participants persist enough log state to recover after a crash.
• A read-only participant can vote read-only and leave before phase two.

Coordinator                 Inventory DB                 Payment DB
    | BEGIN global G-18           |                           |
    | do work -------------------> |                           |
    | do work ----------------------------------------------> |
    | PREPARE? ------------------> |                           |
    |                             | force PREPARED to log     |
    | <---------------------- YES |                           |
    | PREPARE? ---------------------------------------------> |
    |                                                         | force PREPARED
    | <------------------------------------------------- YES  |
    | force COMMIT decision to coordinator log                |
    | COMMIT G-18 --------------> |                           |
    | COMMIT G-18 ------------------------------------------> |
    | <---------------------- ACK |                           |
    | <------------------------------------------------- ACK  |
    | forget completed G-18       |                           |

If any participant votes NO before the commit decision, the coordinator records
ABORT and tells every prepared participant to roll back.

Why prepare is a serious promise

A YES vote means the participant has durably recorded everything needed to commit later and has reserved the resources required to honor that promise. It can no longer unilaterally abort just because its client disconnected. Depending on the resource, prepared state can retain row or metadata locks, transaction identifiers, undo information, and storage. This is why preparation must be short-lived and operationally visible.

Coordinator and participant failure mechanics

Classic 2PC is blocking for a prepared participant that cannot learn the decision. Replicating the coordinator reduces its failure probability but does not turn an unsupported participant into a transactional one. Consensus and 2PC solve different decisions: consensus can replicate the coordinator's state among coordinator nodes, while 2PC obtains one atomic outcome across resource managers that executed different work.

Concrete PostgreSQL operational example

BEGIN;
UPDATE inventory SET reserved = reserved + 1 WHERE sku = 'SKU-8';
PREPARE TRANSACTION 'order-O-701-inventory';

-- Later, from any session after the coordinator decides:
COMMIT PREPARED 'order-O-701-inventory';

-- Or, if the global decision is abort:
ROLLBACK PREPARED 'order-O-701-inventory';

PostgreSQL stores prepared transaction state durably, and a later session completes it. Production use requires a deliberately configured prepared-transaction limit, a durable coordinator, unique IDs, monitoring of pg_prepared_xacts, an age alert, and a runbook that maps each ID to a verified coordinator decision. An operator must not guess from application status alone. PREPARE TRANSACTION is unrelated to a prepared SQL statement created by PREPARE.

When 2PC is reasonable and when it is not

Do not reject 2PC by slogan. First ask whether one database transaction can preserve the invariant. If not, determine whether every participant truly supports atomic commit and whether blocking and coupling fit the workload. A short database-to-broker XA transaction can be appropriate in a controlled platform. A checkout that includes a card network and shipping partner cannot rely on it.

03. Sagas and semantic rollback

A saga preserves a business outcome by committing a series of local steps and executing explicit recovery actions when forward progress becomes impossible.

Compensable, pivot, and retryable steps

• Compensable step: a later command can counteract it, such as release inventory.
• Pivot step: the point after which the workflow commits to forward completion.
• Retryable step: an idempotent action after the pivot that should eventually succeed.

The labels describe business meaning, not an infrastructure feature. For one merchant, payment authorization is compensable because it can be voided. Capturing funds may be the pivot because a refund is financially distinct and incurs fees. Sending a marketing email is neither safely reversible nor important enough to block order confirmation, so it belongs after confirmation as best-effort retryable work.

Design every forward step with its recovery contract

Capture compensation inputs when the forward step commits. Do not reconstruct a past price, address, tax decision, or permission from mutable current data. Make compensation idempotent and conditional on the resource still belonging to this saga. Compensation order is often reverse order, but the domain can require another order or safe parallelism.

Sagas do not provide transaction isolation

While a saga is running, another operation can observe or change its intermediate data. Typical anomalies include lost updates, dirty business reads, and nonrepeatable decisions. Database dirty reads are not required for a dirty business read: a local reservation is committed and visible, but the larger order can still compensate later.

• Semantic lock: mark an entity as pending and restrict conflicting actions.
• Version check: update only if the version seen earlier still matches.
• Escrow or reservation: allocate a bounded right instead of changing final ownership.
• Commutative operation: represent deltas that can safely reorder.
• Reread before pivot: verify price, stock, policy, and customer state again.
• Reorder steps: move high-risk changes closer to the pivot or after it.

04. Orchestration and choreography

Orchestration

An orchestrator stores the workflow state, sends a command for the next step, consumes the result, and chooses the next transition. It offers one readable process definition, explicit deadlines, centralized recovery policy, and a natural place for operator controls. It must be durable and horizontally recoverable, but it need not be a throughput bottleneck: instances can partition workflow IDs while the state store serializes each workflow.

Choreography

In choreography, an order event triggers inventory, an inventory event triggers payment, and later events trigger confirmation or recovery. There is no component issuing every command. This can keep a small, linear flow loosely coupled, but process state becomes implicit in event routes. Cycles, hidden dependencies, duplicated policy, difficult timeouts, and unclear ownership grow quickly as participants and branches increase.

A hybrid is common. The orchestrator controls order fulfillment, while independent analytics and notification consumers react to business events without becoming required workflow participants. Avoid an orchestrator that reads and writes every service database. It should send domain commands through supported contracts; otherwise it removes service ownership and becomes a distributed monolith controller.

05. Durable workflow state machines

Reliability comes from persisted transitions, explicit states, and repeatable decisions, not from keeping one worker or thread alive.

State model

STARTED
  -> RESERVING_INVENTORY
      -> INVENTORY_RESERVED
          -> AUTHORIZING_PAYMENT
              -> PAYMENT_AUTHORIZED
                  -> CONFIRMING_ORDER
                      -> CONFIRMED                    terminal success

Failure before confirmation:
  -> COMPENSATING_PAYMENT
      -> COMPENSATING_INVENTORY
          -> CANCELED                                terminal recovery

Uncertain provider result:
  -> PAYMENT_OUTCOME_UNKNOWN
      -> AUTHORIZING_PAYMENT | PAYMENT_AUTHORIZED | MANUAL_REVIEW

Repeated recovery failure:
  -> COMPENSATION_BLOCKED                            operator-owned

Persist both coarse business state and step attempts. A useful transition row contains workflow ID, current state, version, step name, attempt, command ID, input digest, scheduled time, deadline, result, external reference, error class, and timestamps. Use an optimistic version or a single workflow executor lease with fencing so two workers cannot advance the same instance concurrently.

Atomic transition algorithm

ADVANCE(workflow_id, observed_event):
  begin local transaction
    workflow = lock_or_compare_version(workflow_id)

    if observed_event.id is already consumed:
      return stored transition result

    validate event belongs to expected command and workflow state
    next = transition_table(workflow.state, observed_event.outcome)

    record consumed event ID
    update workflow state, version, business facts, and audit history

    if next has command:
      insert outbox(command_id, workflow_id, next.command, next.deadline)
  commit local transaction

  relay publishes the command later

The state transition, consumed-event marker, and next command intent share one local transaction. A crash before commit leaves all three absent and redelivery retries the same transition. A crash after commit leaves all three present, and duplicate consumption returns the stored result. The relay can publish the command more than once, so the receiving service must deduplicate it.

Timers, retries, deadlines, and cancellation

• Persist wake-up time, never rely only on an in-memory timer.
• Use a monotonic clock for elapsed work inside a process and wall time for durable schedules.
• Classify business rejection separately from transient technical failure and unknown outcome.
• Bound retry attempts, elapsed time, and aggregate retry load; add jitter.
• Use stable command IDs across retries of the same logical effect.
• On cancel, define whether to stop future work, compensate completed work, or reject after pivot.
• Late success after timeout is a valid event that the state machine must handle explicitly.

Replay-based durable execution

Some workflow engines persist a history of decisions, timers, signals, and activity results. After a crash they replay that history through workflow code to reconstruct state, then schedule only new work. Workflow decision code must therefore be deterministic for the recorded history. Wall-clock reads, random values, unordered iteration, network calls, and version-dependent branching must use engine-provided deterministic APIs or activities. Deploy workflow changes with versioning or compatibility markers so old histories still replay.

Durable execution does not make an external payment exactly once. The worker can receive approval, crash before recording it, and execute the activity again. Activity commands still require stable idempotency identity, outcome lookup, heartbeats for long work, and cancellation semantics.

Human intervention is a first-class state

A disputed payment, inconsistent provider records, expired compensation window, or legal approval may require a person. Store MANUAL_REVIEW with reason code, evidence links, permitted actions, deadline, and owner queue. An operator action must be authenticated, authorized, require a comment for sensitive changes, use optimistic concurrency, and create an immutable audit event. Never tell an operator to edit production rows until they look right.

06. Transactional outbox

The outbox converts an unsafe database-plus-broker dual write into one local database commit plus a retryable delivery problem.

Minimal outbox contract

BEGIN;

UPDATE orders
SET status = 'CONFIRMED', version = version + 1
WHERE order_id = 'O-701' AND version = 7;

INSERT INTO outbox_event (
  event_id, aggregate_type, aggregate_id, aggregate_version,
  event_type, schema_version, occurred_at, payload, trace_context
) VALUES (
  'E-9901', 'Order', 'O-701', 8,
  'OrderConfirmed', 3, CURRENT_TIMESTAMP, :payload, :trace_context
);

COMMIT;

The outbox row is immutable event intent. Use a globally unique event ID, aggregate ID as routing key, per-aggregate version, explicit event and schema versions, occurrence time, tenant identity, and a payload that contains the business fact consumers need. Do not publish a database row dump, secret, card data, access token, or unbounded object. A trace context is useful but must not replace business correlation IDs.

Polling publisher

repeat:
  begin transaction
    rows = select due unpublished outbox rows
           ordered by created_at, event_id
           limit batch_size
           lock rows while allowing other relays to skip locked rows
    mark rows with lease_owner and lease_until
  commit

  for row in rows:
    publish(row.routing_key, row.event_id, row.payload)
    if broker acknowledges:
      mark row published if lease_owner still matches
    else:
      release or let lease expire; retry with backoff

Never hold a database transaction open while waiting on a broker unless the design intentionally uses a supported atomic resource transaction. Short leases let another relay recover abandoned rows. A crash after broker acknowledgement but before marking the row published causes a duplicate. That is safe only when consumers are idempotent. Deleting published rows can reduce storage, but keep enough history or archive evidence to support replay, audit, and reconciliation.

Change data capture relay

A change data capture connector reads the database's committed change log and maps inserts from the outbox table to broker messages. This avoids application polling and follows commit order as exposed by the database log. It adds connector, log-retention, schema, offset, snapshot, and operational concerns. If the connector is down, database logs or replication slots can grow until retention limits or disk capacity become incidents.

Debezium's official outbox event router expects insert-oriented outbox records, exposes the unique event ID in a header, and can use the aggregate ID as the message key. That key is important for maintaining per-aggregate order in a partitioned broker. CDC still does not give every consumer one global exactly-once effect. Connector restarts, broker retries, consumer crashes, and downstream side effects retain duplicate windows.

Ordering rules

• State the required scope: per order, per account, per partition, or global.
• Route all events for one aggregate by stable aggregate ID when per-aggregate order matters.
• Include aggregate version so a consumer detects gaps, duplicates, and stale delivery.
• Do not assume created timestamps form a total order across nodes.
• Multiple relay workers can reorder different aggregates safely if no cross-aggregate invariant exists.
• A failed older event must not be skipped silently if later versions depend on it.
• Repartitioning, retries, and dead-letter replays require explicit ordering tests.

07. Inbox, deduplication, and idempotency

At-least-once delivery becomes an effectively-once business effect when duplicate commands are recognized and their effect and receipt are committed atomically.

PROCESS(message):
  authenticate source and validate schema, size, tenant, and expiry

  begin local transaction
    prior = inbox.find(consumer_name, message.id)
    if prior exists:
      commit
      acknowledge message
      return prior.outcome

    assert command identity is valid for message payload digest
    outcome = apply domain transition with database constraints
    insert inbox(consumer_name, message.id, digest, outcome, processed_at)
    insert outbox events caused by outcome
  commit

  acknowledge message

If the worker crashes before commit, redelivery repeats no committed effect.
If it crashes after commit but before acknowledgement, inbox returns prior outcome.

Idempotency key design

• Generate the key at the boundary that knows one logical user intent.
• Scope it by tenant, authenticated principal, operation, and target as appropriate.
• Store a canonical request digest; reject the same key with different parameters.
• Use a unique database constraint to resolve concurrent duplicates atomically.
• Return the original durable outcome, including its business reference.
• Retain keys longer than every client, broker, replay, and disaster-recovery retry window.
• Redact or encrypt stored results that contain personal or financial information.

Deduplication limitations

A finite inbox cannot suppress a duplicate delivered after its record expires. A message ID cannot help if a producer gives each retry a new ID. An in-memory set disappears on restart and diverges across replicas. A broker's duplicate suppression cannot cover a later database or HTTP side effect. Where possible, strengthen idempotency with a domain invariant such as unique payment per (order_id, payment_attempt) or one active reservation per command ID.

08. Exactly-once claims and their boundaries

Exactly once is meaningful only when the operation, scope, state stores, failure model, and observation boundary are named.

Prefer the honest phrase: at-least-once transport with idempotent, deduplicated business effects. Then document the key scope, retention, atomic store, and reconciliation method. This is more useful than claiming global exactly once.

09. Complete order, payment, and inventory design

This design uses an orchestrated saga, service-owned databases, transactional outboxes, idempotent commands, and reconciliation. It optimizes for reliable online checkout without requiring the card network to join a database transaction.

Requirements and assumptions

• An order contains one or more SKUs and one payment method token.
• Peak accepted checkout rate is 2,000 orders per second, with bursts to 4,000.
• Checkout returns an order ID quickly; final confirmation may take several seconds.
• Inventory must never confirm more units than available under normal database correctness.
• A confirmed order must have a valid reservation and payment authorization.
• Inventory reservations expire after 15 minutes unless confirmed.
• Payment provider requests can time out after committing.
• Order, Inventory, and Payment services each own their database and deployment.
• Per-order command and event order matters; global order across all orders does not.
• Customer-visible states are Pending, Confirmed, Canceled, and Needs attention.

Components and ownership

Customer
   |
   | HTTPS + authenticated checkout + Idempotency-Key
   v
API Gateway -> Order API -> Order DB + Outbox
                              |
                              v
                     replicated message broker
                              |
                 Order Workflow Orchestrator
                     |                    |
          ReserveInventory command    AuthorizePayment command
                     |                    |
                     v                    v
             Inventory Service       Payment Service -> tokenized provider API
             Inventory DB+Inbox      Payment DB+Inbox
                     |                    |
                     +------ result events ------+
                              |
                      Orchestrator DB+Outbox

Control plane: operator console -> workflow administration API
Evidence plane: immutable audit sink, metrics, traces, reconciliation reports

Each service database is private. Services communicate through authenticated
contracts, not by querying another service's tables.

Essential records

Happy-path sequence

1. Client -> Order: CreateOrder(checkout key C-12)
2. Order transaction:
     insert order O-701 in PENDING
     insert OrderCreated event E-1
   return 202 Accepted with O-701

3. Orchestrator consumes E-1 transactionally:
     create workflow W-701 in RESERVING_INVENTORY
     insert ReserveInventory command I-CMD-1

4. Inventory consumes I-CMD-1 transactionally:
     verify inbox has no I-CMD-1
     conditionally reserve requested units as reservation R-33
     insert inbox outcome and InventoryReserved event E-2

5. Orchestrator consumes E-2 transactionally:
     store R-33, set AUTHORIZING_PAYMENT
     insert AuthorizePayment command P-CMD-1

6. Payment consumes P-CMD-1:
     use P-CMD-1 as provider idempotency key
     persist APPROVED with provider authorization A-55
     insert PaymentAuthorized event E-3

7. Orchestrator consumes E-3 transactionally:
     store A-55, set CONFIRMING_ORDER
     insert ConfirmOrder command O-CMD-1

8. Order consumes O-CMD-1 transactionally:
     require PENDING and matching workflow
     set CONFIRMED
     insert OrderConfirmed event E-4

9. Orchestrator records CONFIRMED as terminal success.
10. Inventory marks R-33 allocated; later fulfillment and notification react independently.

Confirmation is the pivot in this example. Before it, reservation and authorization are compensable. After it, fulfillment actions must move forward or enter a separately defined return and refund workflow. The design revalidates reservation expiry and authorization validity before confirmation. Notification does not belong on the critical consistency path.

Insufficient inventory path

1. Inventory commits a rejected command outcome and emits InventoryRejected.
2. The orchestrator moves directly to cancel because no previous side effect needs compensation.
3. Order transitions from Pending to Canceled with reason OUT_OF_STOCK.
4. The client sees a durable business rejection, not a retryable technical error.

Payment decline path

1. Payment stores a final declined outcome for P-CMD-1.
2. The orchestrator records the reason, then emits ReleaseInventory(R-33).
3. Inventory changes only reservation R-33 from Held to Released and emits completion.
4. Order becomes Canceled only after release succeeds or enters a visible recovery state.

Unknown payment path

If the provider request times out, Payment records OUTCOME_UNKNOWN; it must not emit Declined. A reconciliation worker queries the provider using the stable key or provider reference. If approved, it stores approval and emits the normal result. If definitively absent, it can retry authorization with the same logical key. If the provider cannot resolve the result before the reservation expires, the workflow enters manual review or follows a preapproved risk policy. Automatically issuing another authorization with a fresh key could double-authorize funds.

End-to-end failure matrix

Recovery and reconciliation jobs

• Stuck workflow sweeper: finds nonterminal workflows past their next wake-up time.
• Outbox auditor: finds unpublished or unacknowledged rows older than the relay objective.
• Reservation sweeper: expires holds and emits an event using the same local transaction.
• Payment resolver: queries unknown provider results by stable payment key.
• Cross-ledger reconciler: compares confirmed orders, reservations, and payment records.
• Provider settlement reconciler: matches provider reports to internal captures and refunds.
• Inbox retention job: removes dedupe records only after the documented redelivery horizon.
• History archival: moves completed workflow and outbox history under retention policy.

Reconciliation must be idempotent, paginated, checkpointed, rate-limited, observable, and safe to resume. It should produce a discrepancy record before repairing. High-impact repair can require approval. Never let a reconciliation job flood an unhealthy provider or silently rewrite financial history.

10. Focused implementation examples

Framework APIs differ, but the correctness pattern stays the same: stable command identity plus one local transaction for domain effect, inbox receipt, and outgoing intent.

Java and Spring-style local transaction

@Transactional
public OrderReceipt create(CreateOrder command, String idempotencyKey) {
    return requests.find(command.customerId(), idempotencyKey)
        .map(RequestRecord::receipt)
        .orElseGet(() -> {
            Order order = orders.insert(Order.pending(command));
            OutboxEvent event = OutboxEvent.orderCreated(order);
            outbox.insert(event);
            requests.insert(command.customerId(), idempotencyKey,
                            command.digest(), order.receipt());
            return order.receipt();
        });
}

@Transactional is only the local database boundary. It does not include a later broker publish. Concurrent duplicate keys need a database unique constraint and conflict handling, not only the initial lookup. Production code validates that a reused key has the same command digest.

Go-style inbox consumer

func HandleReserve(ctx context.Context, msg Message) error {
    return db.WithTx(ctx, func(tx *Tx) error {
        prior, found := tx.Inbox().Find("inventory-reserver", msg.ID)
        if found {
            return nil
        }

        reservation, err := tx.Inventory().Reserve(msg.CommandID, msg.Items)
        if err != nil {
            return err
        }

        tx.Inbox().Insert(msg.ID, msg.Digest, reservation.Result())
        tx.Outbox().Insert(InventoryReserved(msg, reservation))
        return nil
    })
}

// The broker acknowledgement happens only after HandleReserve returns success.

A context deadline stops waiting but does not prove the database rolled back. The transaction helper must resolve commit errors according to the database driver contract. The broker adapter acknowledges only after commit. A duplicate delivery returns success so the broker can remove it.

TypeScript state transition table

const transitions = {
  RESERVING_INVENTORY: {
    InventoryReserved: "AUTHORIZING_PAYMENT",
    InventoryRejected: "CANCELING_ORDER",
  },
  AUTHORIZING_PAYMENT: {
    PaymentAuthorized: "CONFIRMING_ORDER",
    PaymentDeclined: "RELEASING_INVENTORY",
    PaymentUnknown: "RESOLVING_PAYMENT",
  },
  CANCELED: {
    PaymentAuthorized: "VOIDING_LATE_AUTHORIZATION",
  },
} as const;

function decide(state: State, event: Event): Decision {
  const next = transitions[state]?.[event.type];
  if (!next) return quarantine("illegal transition", state, event);
  return persistStateAndOutbox(next, commandFor(next));
}

Real code verifies workflow ID, expected command ID, payload digest, participant identity, and current version. A terminal state is not permission to ignore every late event. A late approval after cancellation creates an obligation to void funds.

11. Production deployment and operations

Deployment model

• Run relays and workflow workers with redundancy across failure domains.
• Partition work by stable workflow or aggregate key, but never depend on one permanent host.
• Use readiness to stop new assignments before shutdown and allow leases to expire safely.
• Keep databases and brokers private; use authenticated, encrypted service connections.
• Back up workflow, inbox, outbox, and coordinator logs consistently with business databases.
• Test restore plus replay, not only backup creation.
• Drain old workflow code only after every compatible history has completed or migrated.

Compatible schema and workflow rollout

Event contracts are long-lived because retained events, delayed messages, and old workflows can outlive a deployment. Add optional fields first, keep readers tolerant of unknown fields, assign schema versions, and test backward and forward compatibility. Never change the meaning of an existing field silently. For a required semantic change, publish a new event version or type and operate old and new consumers during migration.

Workflow state changes need expand-and-contract deployment. First deploy code that reads old and new states, then migrate or naturally advance histories, then remove old handling after evidence shows no old instance remains. Replay-based engines require deterministic version markers. Rolling back application code without considering persisted workflow history can make recovery impossible.

Disaster recovery

Define RPO and RTO for business state and workflow intent together. Restoring an order database to 10:00 and its outbox to 10:05 can publish events for orders that no longer exist. Restoring a broker offset ahead of an inbox can skip required work; restoring it behind can create duplicates. A safe plan names the source of truth, restores compatible checkpoints, expects redelivery, preserves idempotency records, and runs reconciliation before reopening all traffic.

Stuck-workflow runbook

1. Identify scope by state, age, tenant, participant, deployment, and failure domain.
2. Check broker lag, relay age, worker saturation, database health, and provider status.
3. Inspect one workflow history and correlate command ID, event ID, and external reference.
4. Classify the outcome as business rejection, transient, permanent, unknown, or illegal transition.
5. Stop retry amplification or pause the affected step without losing durable messages.
6. Resolve external truth before replaying any ambiguous financial command.
7. Resume, compensate, or send to authorized manual review through a recorded control action.
8. Run scoped reconciliation and monitor the backlog drain rate and new failure rate.
9. Document the cause, affected invariants, customer impact, and permanent prevention.

12. Observability, auditing, and troubleshooting

Observe both transport health and business convergence. A broker can be green while thousands of paid orders remain unconfirmed.

Core metrics

Logs and traces

Structured logs should include workflow ID, order ID, command ID, event ID, causation ID, correlation ID, attempt, current state, next state, tenant, result class, and safe external reference. Propagate trace context through outbox messages, but create spans for queue wait, relay, consumer processing, provider call, and state transition. Sampling must retain errors and slow or blocked workflows. Do not use high-cardinality business IDs as metric labels; keep them in logs and traces.

Audit requirements

• Record who or what requested every financial and inventory action.
• Preserve original and compensating actions as separate immutable facts.
• Record workflow policy and code version used for each decision.
• Timestamp receipt, decision, dispatch, participant completion, and operator action.
• Protect audit logs with access control, integrity monitoring, retention, and legal policy.
• Separate sensitive payloads from widely accessible operational metadata.
• Support a customer or order timeline without exposing secrets or another tenant's data.

Useful alert examples

Page on invariant risk and sustained inability to make progress: old unknown payments, blocked compensation above a value threshold, oldest outbox age over the delivery objective, or a sharp increase in nonterminal workflow age. Ticket lower-urgency duplicate-rate drift or isolated poison messages. An alert on every retry creates noise; an alert on a growing backlog with no drain rate describes user impact.

13. Security, privacy, abuse, and trust boundaries

Authenticate every transition

A broker is a transport, not proof that a message is authorized. Authenticate producers and consumers with workload identity, authorize publish and subscribe permissions by topic or queue, encrypt transport, rotate credentials, and restrict network paths. The participant validates the command against its own business policy and tenant boundary. An orchestrator request is not permission to bypass inventory or payment rules.

Replay and idempotency abuse

• Bind an idempotency key to principal, tenant, operation, resource, and request digest.
• Reject expired messages and impossible future timestamps according to a documented skew window.
• Use unguessable message IDs, but do not treat uniqueness as authentication.
• Protect command envelopes against tampering through authenticated channels or signatures.
• Rate-limit new commands and repeated outcome queries separately.
• Alert when one key arrives with multiple payload digests.
• Do not let a caller choose another tenant's aggregate key or workflow ID.

Data minimization and privacy

Events are copied into outboxes, broker replicas, consumer storage, logs, dead-letter queues, backups, and analytics systems. Publish stable identifiers and necessary facts, not full customer profiles. Use payment tokens, never raw card data. Encrypt sensitive payload fields when required, control key access, set retention per copy, and design deletion or anonymization for immutable event histories. A dead-letter queue needs the same protection as the source payload.

Operator and recovery controls

Separate read, retry, compensate, refund, override, and replay permissions. High-value refunds or heuristic transaction decisions can require two-person approval. Every control request needs a reason, ticket, actor, before-state, after-state, and idempotency key. Guard bulk replay by tenant, date, event type, maximum count, and dry-run preview. Audit access to customer and payment evidence.

Threat and mitigation table

14. Performance, capacity, scalability, and cost

Count the durable work

A three-step saga does more work than one local transaction. Each command can add an outbox insert, broker record, inbox insert, domain update, result outbox insert, workflow transition, indexes, and replicated log writes. If one order produces 8 messages and traffic peaks at 4,000 orders per second, the broker sees about 32,000 business messages per second before retries and replicas. At an average 2 KB payload, raw ingress is about 64 MB/s before protocol and replication overhead.

Latency budget

Order local commit + relay:          120 ms
Workflow consume + reserve command:  80 ms
Inventory process + result:         180 ms
Workflow consume + payment command:  80 ms
Payment provider authorization:     700 ms
Workflow consume + confirm command:  80 ms
Order confirmation + final event:   160 ms
Scheduling and variance reserve:    600 ms
                                      -------
Target p95:                         2,000 ms

Measure queue wait separately from processing. Batching improves throughput but increases the wait for the oldest event. Poll intervals directly add outbox delivery latency. Long synchronous client waits consume connections, so return a durable order ID and support polling, push notification, or a bounded synchronous wait followed by Pending.

Capacity and overload controls

• Bound workflow activation, relay batches, in-flight provider calls, and per-tenant concurrency.
• Partition by aggregate ID to preserve order and distribute load; watch hot tenants and hot SKUs.
• Use backpressure from consumer lag and database saturation before queues become unbounded.
• Reserve capacity for compensation, outcome resolution, and operator actions during incidents.
• Apply retry budgets so provider failure does not multiply every pending workflow attempt.
• Use payload references for very large objects, with integrity and authorization checks.
• Scale relays only until the database or broker becomes the bottleneck.

Storage and retention

At 32,000 messages per second and 2 KB each, one raw day is roughly 5.5 TB before replication, indexes, and compression. Real retention design separates broker replay retention, immutable audit, compact workflow summaries, inbox dedupe horizon, and outbox cleanup. Partition high-volume tables by time where supported, archive completed history, and test deletion without blocking live writes. Retaining everything in the primary workflow database is expensive and slows maintenance.

Cost trade-offs

Main cost drivers are replicated broker storage, database write amplification, cross-region traffic, workflow history, CDC infrastructure, provider outcome queries, and operator effort. Orchestration adds a durable store but can reduce incident diagnosis time. Choreography removes one coordinator service but can increase observability and change-coordination cost. 2PC may reduce compensation logic for a compatible short operation, while increasing latency and lock risk.

15. Testing strategy

Unit and model tests

• Test every allowed state-event pair and every illegal transition.
• Test compensation choice for failures before, at, and after the pivot.
• Test retry classification: business rejection, transient, permanent, and unknown.
• Test idempotency key scope, digest mismatch, and concurrent duplicate decisions.
• Use property tests: terminal success implies payment plus reservation; cancellation eventually releases both.
• Model-check small histories with duplicate, lost, late, and reordered messages.

Integration and contract tests

• Commit business state and outbox atomically in the real database.
• Crash after publish and before marking published; verify consumer effect occurs once.
• Crash consumer before commit and after commit before acknowledgement.
• Run two consumers concurrently with the same command ID and verify one domain effect.
• Validate event schemas against old and new producer and consumer versions.
• Test CDC connector restart, offset recovery, snapshot behavior, and log retention.
• For 2PC, kill coordinator and participant at every protocol boundary and recover in-doubt records.

Fault, recovery, and chaos tests

• Inject delay, loss, duplication, reordering, partition, process crash, and clock skew.
• Return provider success but drop its response to create an ambiguous payment.
• Pause the orchestrator longer than a reservation lifetime and validate late-event policy.
• Make compensation fail repeatedly and verify escalation without infinite retry.
• Restore from backup with broker redelivery and verify inbox retention protects effects.
• Replay a dead-letter range in dry-run and live modes with rate controls.
• Prove reconciliation detects deliberately injected cross-service mismatches.

Load and soak tests

Measure accepted orders, completion throughput, workflow duration percentiles, outbox age, broker lag, database transaction latency, provider concurrency, duplicate rate, and compensation capacity. Stress until admission control activates. Soak beyond inbox cleanup, reservation expiry, log rotation, CDC checkpoint, and partition maintenance intervals. Include a provider slowdown because stable average traffic can still build an unbounded pending-work backlog.

Correctness oracle

For every terminal workflow:
  if state == CONFIRMED:
    assert exactly one valid payment authorization
    assert exactly one allocated inventory reservation
    assert order status == CONFIRMED

  if state == CANCELED:
    assert no active reservation remains
    assert no unvoided or unrefunded payment remains

For every nonterminal workflow older than its objective:
  assert an automatic wake-up or a named operator owner exists

16. Common mistakes and safer corrections

17. Decision framework

1. Write the exact business invariant and acceptable intermediate states.
2. Name every durable resource and the owner of each write.
3. Ask whether ownership can be simplified into one local transaction.
4. If not, list which resources support a common atomic commit protocol.
5. Decide whether blocking, latency, participant coupling, and recovery fit 2PC.
6. For a saga, classify every step as compensable, pivot, retryable, or optional.
7. Define stable command identity and outcome-query behavior for ambiguous effects.
8. Choose orchestration, choreography, or a hybrid based on process complexity and ownership.
9. Close every database-message dual write with outbox; close consumer duplicates with inbox.
10. Define ordering scope, versioning, deadlines, late events, and cancellation.
11. Reserve capacity and operator paths for compensation and reconciliation.
12. Test failures at every commit, publish, acknowledgement, and external-call boundary.

18. Hands-on exercises and design scenarios

Exercise 1: Find the dual-write window

A notification service inserts a job, commits, publishes its ID, then marks the job queued. Draw a crash at each boundary and redesign it. Expected reasoning: place job and outbox intent in one transaction; relay with at-least-once delivery; consume with an inbox or a unique notification key; model sending as Unknown if the provider response is lost.

Exercise 2: Classify travel steps

Design flight hold, hotel booking, card capture, ticket issue, and email. Identify compensation and pivot. Expected reasoning: holds are naturally compensable; booking cancellation may have rules; capture could be pivot or compensable by refund depending on the business; ticket issue may be irreversible; email is after the core outcome. Store cancellation policy and original references.

Exercise 3: Late event

PaymentAuthorized arrives 20 minutes after inventory expired and order canceled. Expected reasoning: verify command identity, record the late event, issue an idempotent void or refund, preserve canceled order status, alert if financial recovery fails, and add reconciliation coverage.

Exercise 4: Polling versus CDC

Compare a system with 50 events per second and a simple operations team to one with 100,000 events per second and an existing CDC platform. Expected reasoning: polling is simpler and can be enough at modest scale; CDC reduces query polling and follows the commit log but needs offset, log-retention, schema, snapshot, and connector operations. Both need idempotent consumers.

Exercise 5: Capacity during provider outage

Checkout continues at 2,000 per second while payment is unavailable for 30 minutes. Expected reasoning: 3.6 million workflows accumulate, reservations can exhaust stock, retry traffic can amplify load, and recovery needs provider capacity above arrival rate. Introduce admission policy, retry budgets, reservation limits, visible Pending status, backlog storage sizing, and reserved recovery capacity.

Exercise 6: 2PC recovery drill

A coordinator crashes after both databases prepare but before one receives commit. Expected reasoning: prepared participants must not guess; recover the durable coordinator decision, resend idempotent commit, monitor prepared age and locks, and use a heuristic decision only under an exceptional documented policy that acknowledges possible atomicity violation.

19. Interview questions and model answers

1. Why is writing to a database and then a broker unsafe?

They are two independent durable commits. A crash between them leaves business state without its event. Reversing the order can leave an event for state that never committed. Use one supported distributed transaction or commit an outbox row beside business state and relay it later. Follow-up: Can the relay publish twice? Yes, after publish succeeds but before the relay records success. Consumers must be idempotent.

2. Explain two-phase commit and why it can block.

In phase one, each participant durably promises it can commit. If all vote yes, the coordinator durably chooses commit; otherwise it chooses abort. In phase two it announces the decision. A prepared participant cannot safely choose on its own when the coordinator decision is unavailable, so it can hold resources while waiting. Follow-up: Does replication fix it? Replicating coordinator state improves availability, but participants still need protocol support and can be unable to learn a decision during a partition.

3. Is a saga an ACID transaction?

No. Each step can be a local ACID transaction, but the whole saga usually exposes committed intermediate state and reaches consistency through forward actions or compensation. It needs explicit concurrency controls for saga isolation anomalies. Follow-up: Name one control. Use a reservation, semantic pending state, version check, or reread before pivot.

4. What is a compensating transaction?

It is a new domain action that counteracts an earlier committed effect. It is not a byte-for-byte rollback. Refund and charge both remain in the ledger, and concurrency may prevent restoration of the exact previous state. Compensation itself must be idempotent, durable, retryable where safe, and auditable. Follow-up: Must compensation run in reverse order? Often, but domain dependencies can require another order or allow parallel steps.

5. Orchestration or choreography?

Choreography is reasonable for a few stable reactions and maximizes participant autonomy. Orchestration makes complex branches, timers, compensation, visibility, and operator controls explicit. A hybrid often orchestrates the critical business flow and choreographs optional analytics and notifications. Follow-up: Is the orchestrator a single point of failure? Only if state is not durable or the deployment has one unrecoverable instance. Its unavailability can delay progress without losing it.

6. What is a pivot transaction?

It is the point of commitment after which the saga should move forward rather than unwind earlier compensable steps. The exact pivot is a business decision. Before it, the design emphasizes compensation; after it, actions should be idempotent and retryable. Follow-up: Give an example. Order confirmation can be the pivot after inventory and payment authorization.

7. How does an inbox create an effectively-once effect?

The consumer atomically stores the message ID, applies its domain effect, and writes any outgoing event. Redelivery finds the stored receipt and does not repeat the effect. This needs a unique constraint, stable producer identity, sufficient retention, and one local transaction. Follow-up: Why is an in-memory set insufficient? It is lost on restart and is not shared atomically with durable business state.

8. What should a service do after a payment timeout?

Record Unknown, preserve the stable payment key, and query or retry using that same identity. A timeout does not prove failure. Reconciliation compares provider records and internal state. A new key can create a second authorization. Follow-up: When do you involve a human? When automated evidence remains ambiguous past the risk deadline or the amount exceeds policy.

9. Does transactional outbox guarantee event order?

It guarantees atomic presence of business state and event intent. Ordering depends on database log order or polling selection, relay concurrency, broker partitioning, retries, and consumer behavior. Use an aggregate routing key and aggregate version for per-aggregate order. Global order is costly and usually unnecessary. Follow-up: What does a consumer do on a version gap? Pause that aggregate and fetch, replay, or rebuild rather than silently apply dependent later state.

10. What does exactly once mean in a distributed workflow?

The statement is incomplete until its scope is named. A broker may atomically commit input offsets and output messages, while an external HTTP effect remains outside. Durable workflow replay can preserve decisions, while an activity can repeat after its result is lost. Prefer at-least-once delivery with idempotent business effects and reconciliation. Follow-up: What breaks dedupe? New keys, expired records, non-atomic receipt storage, or effects outside the dedupe store.

11. How do you monitor a saga system?

Track completion and cancellation rates, duration percentiles, oldest nonterminal workflow, outbox age, broker lag, duplicate hits, unknown outcomes, failed compensation, and reconciliation mismatches. Correlate logs and traces with workflow, command, event, and external reference IDs. Alert on sustained failure to converge, not every retry. Follow-up: Why is broker lag insufficient? The broker can be current while a provider result remains unknown or business state is inconsistent.

12. When would you choose one database over a saga?

When the data has one cohesive owner, needs one atomic invariant, and can scale and operate within the same transactional store. Distribution adds failure states, messages, dedupe, compensation, and operator burden. Service boundaries should follow business ownership, not a rule that every table needs a service. Follow-up: When must you distribute? Independent ownership, scaling, regulatory isolation, or external participants can justify it.

13. How do you safely change a running durable workflow?

Keep event contracts backward compatible, make code read old and new state, introduce new states before writing them, and retire old handling only after no old history remains. Replay-based engines need deterministic version markers. Follow-up: Why can normal rollback fail? Persisted histories may contain events that the older code cannot interpret or replay.

14. What happens if compensation fails?

Compensation is another distributed workflow. Persist its progress, retry only safe transient failures with limits, retain the original recovery inputs, and move persistent or ambiguous cases to a visible blocked state with an owner. Reconciliation must continue checking the obligation. Follow-up: Should the system report Canceled first? Only if the API distinguishes cancellation requested from fully compensated; never hide an active charge or reservation.

20. Revision cheat sheet

21. Production review checklist

• The business invariant, atomic boundary, and acceptable pending states are written.
• Every forward step has retry, timeout, unknown-outcome, and compensation behavior.
• The pivot and every irreversible side effect are explicit.
• Business change and outgoing event intent share one transaction.
• Consumer effect, inbox receipt, and resulting outbox share one transaction.
• Command keys are stable, scoped, digest-bound, retained, and unique.
• Per-aggregate ordering and version-gap behavior are tested.
• Timers and progress survive process, zone, and deployment failure.
• Late results and cancellation races have state transitions.
• Compensation has reserved capacity and a manual escalation path.
• Unknown external results have query and reconciliation paths.
• Schema and workflow upgrades remain compatible with retained history.
• Metrics expose oldest pending intent and business convergence.
• Messages, logs, dead letters, backups, and audits follow privacy policy.
• Restore, redelivery, replay, and reconciliation are exercised regularly.

22. Primary official references

• Jakarta Transactions 2.0 Specification - transaction managers, resource enlistment, XAResource, synchronization, and two-phase commit.
• Java SE XAResource API - prepare, commit, rollback, recovery scanning, transaction branches, and heuristic outcomes.
• PostgreSQL: PREPARE TRANSACTION - durable preparation, later completion, warnings, and two-phase transaction behavior.
• PostgreSQL: pg_prepared_xacts - official catalog view for monitoring currently prepared transactions.
• PostgreSQL: max_prepared_transactions - prepared-transaction capacity configuration and operational cautions.
• Microsoft Azure Architecture Center: Saga distributed transactions pattern - local transactions, orchestration, choreography, pivot and retryable steps, and anomalies.
• Microsoft Azure Architecture Center: Compensating Transaction pattern - semantic undo, idempotent compensation, progress recording, and manual intervention.
• AWS Prescriptive Guidance: Saga orchestration pattern - coordinator responsibilities, applicability, local steps, and failure handling.
• AWS Prescriptive Guidance: Transactional outbox pattern - dual-write avoidance, relay options, ordering, and duplicate delivery.
• Debezium: Outbox Event Router - outbox schema mapping, unique event IDs, aggregate routing keys, payloads, and tracing.
• Apache Kafka official documentation: delivery semantics - at-most-once, at-least-once, transactions, idempotent producers, and exactly-once boundaries.
• CloudEvents 1.0.2 Specification - interoperable event context attributes, event identity, source, type, time, and data contracts.
• OpenTelemetry semantic conventions: messaging spans - producer, consumer, creation, settlement, and messaging trace relationships.
• Temporal official documentation: Workflows - durable workflow execution, event history, deterministic replay, timers, and signals.