Architecture Styles, Modular Monoliths, Microservices, and Service Boundaries - Complete Notes

A language-neutral guide to choosing architecture constraints, finding cohesive business boundaries, building a disciplined modular monolith, distributing only when the evidence justifies it, and evolving an order platform without creating a distributed monolith.

00. Mental model and precise terminology

Architecture is the set of boundaries and dependency rules that make change safe enough for a system's expected lifetime.

01. The problem architecture solves

The goal is not the maximum number of boxes. The goal is sustainable correctness under change, load, failure, and team growth.

Every system begins with assumptions. This guide's running example is an order platform with catalog browsing, checkout, payment authorization, inventory reservation, fulfillment, and notification. Assume 300 requests per second at normal peak, 1,500 requests per second during campaigns, 30 checkout commands per second, a team growing from 8 to 45 engineers, a requirement to protect payment data, and a need to change catalog features much more often than fulfillment. These facts may justify different boundaries later. They do not justify starting with six remote services on day one.

Forces a design must balance

• Correctness: where invariants and transactions can be enforced.
• Change: which concepts evolve together and which teams need autonomy.
• Runtime: latency, throughput, scaling shape, isolation, and availability.
• Operations: deployments, alerts, debugging, recovery, and on-call ownership.
• Security: identities, trust boundaries, secrets, data classification, and audit.
• Economics: engineering time, infrastructure, licenses, data transfer, and waste.
• Reversibility: how safely a wrong boundary can be moved, joined, or split.

observe change patterns, load, incidents, and ownership friction
                         |
                         v
identify one painful dependency or failure domain
                         |
                         v
propose the smallest boundary rule that addresses it
                         |
                         v
measure delivery speed, correctness, latency, cost, and recovery
                         |
             keep, adjust, merge, or split

02. Cohesion, coupling, and dependency direction

Cohesion means belonging together

Place behavior together when it enforces the same business rules, uses the same language, changes for the same reasons, and has one accountable owner. Checkout price calculation and order-total validation are cohesive. Sending a marketing email merely happens after checkout and belongs to a different capability. Temporal sequence alone does not create cohesion.

Coupling has several forms

Loose coupling does not mean no knowledge. A consumer must know a useful contract. The goal is to depend on a small, stable meaning instead of volatile implementation details. Prefer dependencies that point toward stable business policies. Make forbidden dependencies mechanically impossible where practical.

03. Layered and N-tier architecture

Layers separate kinds of work; tiers place some layers in separate runtime or network locations.

HTTP / UI layer       parses transport, authenticates, formats response
        |
application layer    coordinates use cases and transaction boundaries
        |
domain layer         owns business concepts, policies, and invariants
        |
data access layer    maps durable state through defined interfaces

A closed layer calls only the next layer. An open layer may skip levels, which can reduce ceremony but weakens isolation. A three-tier deployment might run web, application, and database tiers on different hosts. A single process can still have four logical layers. Network placement changes failure and latency; logical layering alone does not.

Layered architecture suits conventional business systems with stable workflows and one coordinated release. It becomes painful when horizontal technical teams own presentation, logic, and data separately, because every feature crosses three queues. Vertical capability ownership usually reduces handoffs.

04. Hexagonal and clean architecture

Business policy stays in the center; technology reaches it through ports implemented by adapters.

inbound adapters                     outbound adapters
HTTP controller ---\               /--- PostgreSQL repository
queue consumer ------> use-case ports <---- payment HTTP client
CLI command --------/      |        \--- event publisher
                       domain model

Dependency rule: adapters depend inward; the domain does not import adapters.

An inbound port expresses what the application can do, such as PlaceOrder. An outbound port expresses what the application needs, such as ReserveInventory or OrderRepository. HTTP, a queue, PostgreSQL, and a payment gateway are adapters. Clean architecture describes a similar inward dependency rule with concentric policy boundaries. The names differ across literature, but the useful test is concrete: can domain policy run in a unit test without a server, database, broker, or framework container?

interface PlaceOrder:
    execute(command, request_id) -> PlaceOrderResult

interface OrderStore:
    find(order_id) -> Order?
    save(order, expected_version)

execute(command, request_id):
    existing = requestLog.find(request_id)
    if existing: return existing.result
    order = Order.create(command.lines, pricingPolicy)
    orderStore.save(order, expected_version = NONE)
    outbox.append(OrderPlaced.v1(order.summary))
    return accepted(order.id)

The pattern improves testability and protects policy from volatile infrastructure. It does not justify an interface for every class. Add a port where ownership, volatility, test substitution, or a real technology boundary exists. Too many one-method wrappers hide the actual flow and create maintenance without isolation.

05. Modular monoliths

A modular monolith keeps local-call simplicity while treating capability boundaries as real architecture.

order-platform process
+---------------------------------------------------------------+
| catalog     ordering      payments      inventory      notify |
| public API  public API    public API    public API     API     |
| internal    internal      internal      internal       internal|
| tables      tables        tables        tables         tables  |
+---------------------------------------------------------------+
         in-process commands and published module events

One artifact, one rollout, one runtime failure domain
Separate models, APIs, schema ownership, and code ownership

Rules that make it genuinely modular

1. Organize top-level code by business capability, not only by technical layer.
2. Each module publishes a small API. Internal packages are inaccessible to other modules.
3. A module is the only writer of its tables. Cross-module reads use APIs or explicit projections.
4. Cross-module calls point one way or pass through an application workflow to avoid cycles.
5. Module events are immutable facts with documented semantics, even when delivered in process.
6. Architecture tests fail builds on forbidden imports or schema access.
7. Ownership, telemetry, and change history are reported by module.

Local transactions are an advantage

If ordering and inventory still share one transactional database, one local transaction may enforce a truly local cross-module invariant. Use that advantage deliberately. Do not pretend a network exists by serializing every internal call. Instead, record which transaction crosses a proposed boundary. That list is evidence about whether the boundary is wrong or whether a later split will require a saga, reservation, or changed invariant.

// ordering/public.ts
export type PlaceOrder = Readonly<{
  requestId: string;
  customerId: string;
  lines: ReadonlyArray<{ sku: string; quantity: number }>;
}>;

export interface OrderingApi {
  place(command: PlaceOrder): Promise<{ orderId: string; status: "accepted" }>;
}

// inventory/internal/stock-row.ts is not exported.
// Ordering receives availability through InventoryApi, never by importing StockRow.

Scaling and deployment

A monolith can run many stateless replicas behind a load balancer. It can have queue workers built from the same repository and database replicas for read load. Its constraint is deployment and runtime granularity, not a single machine. The main inefficiency is coarse scaling: if image rendering uses 80 percent of CPU but ordering uses little, every full replica contains both. First consider moving rendering to a worker process. A service split is only one option.

When it is usually the best starting point

• The domain is still being discovered and boundaries are likely to move.
• One or a few teams can coordinate a release without material delay.
• Most workflows need simple local transactions and low latency.
• The whole workload fits one practical scaling and availability envelope.
• The organization does not yet have mature service ownership, telemetry, and deployment automation.

06. Microservices

Microservices trade local simplicity for independent change, ownership, scaling, and failure isolation across carefully chosen business boundaries.

Useful constraints

• One accountable team owns the service from design through on-call recovery.
• The service models a cohesive business capability in its own language.
• It exposes explicit network contracts and hides implementation details.
• It controls writes and migrations for its data.
• It can be built, tested, deployed, rolled back, and scaled independently.
• It tolerates downstream latency, duplicate delivery, partial failure, and version skew.

These properties matter more than line count. A 70,000-line pricing service can be a coherent microservice. Twenty 1,000-line CRUD services sharing tables and a release train form a distributed monolith. Independence is an observed capability, not a naming convention.

The distribution tax

07. Service-oriented architecture

Service-oriented architecture, or SOA, is the broad idea of exposing business capabilities through interoperable service contracts. Traditional enterprise SOA often uses fewer, coarser services, centralized governance, shared canonical messages, and integration middleware. Microservices are one more decentralized service-oriented style, emphasizing bounded business ownership, independent deployment, decentralized data, and team autonomy.

These are tendencies, not definitions. A centralized transformation bus can become a bottleneck when it owns business behavior that no domain team owns. Equally, duplicating routing, security, and telemetry in every microservice wastes effort. Centralize platform capabilities that are truly uniform, but keep domain decisions with the team and service that own the domain.

08. Event-driven architecture

Producers publish facts about completed state transitions; consumers react without a direct request-time dependency on the producer.

Ordering transaction
    |-- save order
    '-- save OrderPlaced.v1 to outbox
                 |
                 v
              broker
            /    |     \
     inventory  analytics  notification
       reserve    count       email

Producer does not wait for every consumer.
Each consumer owns offset, retry, deduplication, and recovery.

Events can improve temporal decoupling, fan-out, buffering, and independent consumer scaling. They introduce eventual consistency, duplicate delivery, ordering scopes, replay, schema evolution, poison messages, and harder end-to-end reasoning. A command states an intent for one owner, such as ReserveStock. An event states an observed fact, such as InventoryReserved. Naming a command StockShouldBeReserved and broadcasting it does not make it an event.

Event coupling still exists

• Semantic coupling: consumers interpret the producer's meaning.
• Schema coupling: consumers parse fields and types.
• Ordering coupling: a consumer may require events for one order in sequence.
• Availability coupling: recovery may require broker retention and replay.
• Volume coupling: a producer traffic spike becomes consumer lag.

Publish stable business facts, include event identity, subject identity, occurrence time, schema version, trace context, and producer identity, and avoid exposing internal database rows. Use an outbox when state and event publication must agree. Consumers must be idempotent and able to rebuild projections from a documented starting point or snapshot.

09. Web-queue-worker

A web front end handles interactive requests, places long or resource-heavy work on a durable queue, and one or more workers process it outside the request. It is excellent for a relatively simple domain with image conversion, report generation, email delivery, imports, or batch work.

client -POST /files-> web -store metadata + outbox-> database
  |                         |
  |                         '--publish FileProcessingRequested-> queue
  '<-202 Accepted + job URI                                      |
              |                                                  v
              '--GET /jobs/{id} <-- status store <-- worker converts file

Return 202 Accepted only after durable acceptance. Give the client a job identifier and status resource. Bound message size, execution time, retries, and queue age. Workers acknowledge only after durable success, extend visibility carefully for long jobs, and tolerate redelivery. Route permanent failures to quarantine with a reason and operator workflow. Autoscale from queue age and processing time, not depth alone.

10. Serverless architecture

Serverless means the platform assumes more responsibility for provisioning, scaling, patching, and charging for execution or consumed capacity. Functions, managed queues, event routers, databases, and workflows are common building blocks. Code still runs on servers, and the team still owns correctness, security, configuration, dependencies, limits, telemetry, and cost.

Consider cold-start tail latency, maximum duration and payload, concurrency quotas, regional availability, runtime version, observability gaps, egress, and per-invocation pricing. A long-lived high-throughput service may cost less and behave more predictably on continuously provisioned compute. Serverless changes the operating model, not the need for coherent service boundaries.

11. Comparing and composing styles

Turn architectural intent into fitness functions

A fitness function is an automated or routinely reviewed check that tells whether the system still has a desired property. Examples include no import from another module's internal namespace, no cross-service database grant, backward-compatible API schemas, a maximum checkout dependency count, independent deploy success rate, per-service recovery objectives, and a latency budget. Architecture diagrams that disagree with these checks are historical illustrations, not controls.

12. Discovering service boundaries

Find boundaries from business language, invariants, change patterns, ownership, and runtime needs, then validate them with real workflows.

Decompose by business capability

A business capability is an outcome the organization must perform, such as price merchandise, accept an order, authorize payment, reserve stock, or arrange shipment. Capability boundaries are more stable than user-interface screens and database entities. A checkout page touches catalog, promotions, ordering, payments, and inventory; making the page one service would couple unrelated models. A CustomerService, OrderService, and OrderLineService split derived from nouns often creates chatty CRUD services with weak behavior.

Bounded contexts and context mapping

The word product means sellable description and price in Catalog, an SKU and available quantity in Inventory, and a historical line snapshot in Ordering. Forcing one enterprise Product model couples all three. Let each bounded context own its model and translate at the boundary. Record relationships such as upstream/downstream, customer/supplier, conformist, shared kernel, published language, or anti-corruption layer. The label matters less than making influence and translation explicit.

Aggregates define immediate consistency

An Order aggregate might enforce positive quantities, one currency, and a valid status transition. Inventory stock is a different aggregate because global stock contention and reservation lifecycle differ. A service may own many aggregates. Avoid a service per aggregate, which creates remote calls for ordinary behavior, and avoid one aggregate spanning services, which makes its invariants impossible to enforce locally.

A repeatable boundary workshop

1. List actors, commands, business events, policies, read models, and external systems.
2. Draw one end-to-end timeline, including declines, timeouts, cancellation, and recovery.
3. Group facts and rules that share language, invariants, and experts.
4. Mark every synchronous dependency and every transaction crossing a proposed group.
5. Overlay code change history, team ownership, load, data classification, and incidents.
6. Challenge every remote boundary by modeling it as an in-process module first.
7. Document decision, assumptions, rejected alternatives, measures, and a review date.

13. Domain, team, and data ownership

A service boundary needs one team able to make and operate changes. Shared ownership often means no ownership during an incident. Team boundaries should follow capabilities where possible, while a platform team supplies paved roads for build, deployment, identity, telemetry, and messaging. A platform is a product with users and support, not a gatekeeping committee.

Database per service is an ownership rule

The rule does not require one physical database server per service. Separate databases, schemas, or accounts can share a cluster if permissions, migrations, resource limits, backup objectives, and ownership remain clear. Only the owner writes its data. Other services call its API, consume its events, or maintain a local projection. The physical arrangement can evolve without changing that contract.

Shared-database risks

• A consumer silently depends on private columns, indexes, or transaction timing.
• One migration breaks many services and forces coordinated releases.
• Cross-schema joins bypass authorization and business invariants.
• Long queries, locks, or connection spikes create a shared failure domain.
• Ownership of backup, retention, deletion, and incident response becomes unclear.
• A compromised service account gains data unrelated to its capability.

A legacy shared database can be an explicit transitional state. Establish table ownership, revoke new cross-owner writes, place owner APIs before tables, log remaining readers, build projections, and migrate one dependency at a time. Pretending the dependency does not exist makes extraction riskier.

14. Synchronous and asynchronous communication

Do not replace every call with an event. A product-detail request needs a response. Do not make every step synchronous either. Email delivery need not reduce checkout availability. Trace the user promise, decide which facts must be known before replying, and move only deferrable work off the critical path.

15. Gateways, BFFs, discovery, meshes, and sidecars

Infrastructure can standardize communication mechanics, but it must not become the hidden owner of domain behavior.

API gateway

A gateway is an entry point that routes external traffic and may terminate TLS, authenticate, enforce coarse authorization, limit rates, validate payload bounds, translate protocols, and emit edge telemetry. It can hide internal topology and support gradual routing during migration. Keep checkout policy, price calculation, and workflow state out of generic gateway configuration.

Backend for frontend

A backend-for-frontend, or BFF, is a client-specific edge service. A mobile BFF may reduce round trips and payload size, while an operations BFF assembles richer administration data. It should adapt and compose, not become the only home of business rules. Use separate BFFs only when client needs and ownership differ materially. Otherwise one API creates less duplication.

mobile app -TLS-> edge gateway -> mobile BFF -> Ordering logical service
                                                |
                                  discover healthy endpoints
                                                |
                                client or proxy load balances
                                                |
                                   Ordering instance + sidecar

Gateway: north-south policy and routing
Service mesh: optional east-west transport policy and telemetry
Application: business authorization, deadlines, and domain behavior

Service discovery

Instances are ephemeral, so callers need a stable logical name mapped to changing healthy endpoints. Server-side discovery sends traffic to a stable proxy or virtual address. Client-side discovery gives the caller an endpoint set and balancing policy. DNS and registries are common. Discovery is normally eventually consistent, so clients still need connection timeouts, health signals, endpoint draining, and retry rules. A stale endpoint must be an expected condition.

Service mesh and sidecars

A service mesh places service-to-service networking under a common data plane, often proxies near workloads, controlled by a separate configuration plane. A sidecar is a companion process or container sharing a workload lifecycle. It can provide mTLS, identity, discovery, balancing, retries, traffic policy, and telemetry consistently across languages.

A mesh adds proxy hops, CPU and memory, certificate and policy lifecycles, control-plane failure modes, configuration propagation delay, and another place where retries can amplify traffic. It cannot infer business idempotency, choose a correct deadline, authorize an order transition, or make a non-transactional workflow atomic. Adopt it when fleet scale and language diversity repay its operation, not merely because services exist.

16. Contracts, versioning, and compatibility

Independent deployment is possible only when producers and consumers tolerate version skew.

A contract is more than a schema

• Operation and resource meaning, preconditions, and authorization.
• Field type, optionality, units, defaults, and validation limits.
• Success, error, retry, idempotency, timeout, and cancellation semantics.
• Ordering, delivery, retention, and replay rules for messages.
• Rate, payload, concurrency, latency, and availability expectations.
• Deprecation, support window, ownership, and incident contact.

Expand, migrate, contract

1. Producer adds an optional field or new operation without changing old meaning.
2. Producer accepts both old and new request shapes.
3. Consumers migrate independently and report old-version traffic.
4. Owner announces deprecation and waits through the agreed support window.
5. Producer removes old behavior only after evidence shows no authorized consumer uses it.

Favor additive evolution. Consumers should ignore unknown fields where the format permits and not infer meaning from undocumented enum exhaustiveness. Producers should not repurpose fields or silently change units. A new major version is necessary when meaning cannot remain compatible, but major versions also create parallel support cost. Version a contract, not every deployment.

Consumer-driven contracts

Each consumer records the interactions and fields it relies on. Provider verification runs those expectations against the candidate provider. This reveals whether a provider change breaks known consumers without requiring a full shared environment. It does not prove business correctness, performance, security, or behavior of unregistered consumers. Providers remain responsible for an authoritative contract and for preventing consumers from demanding contradictory internals.

OrderPlaced.v1
event_id, occurred_at, order_id, customer_id, total_minor, currency

Safe additive evolution:
+ optional sales_channel with documented default "unknown"

Unsafe in-place evolution:
- total_minor changes from integer minor units to decimal major units
- currency disappears
- event changes from "accepted order" to "paid order"

Changed meaning requires a new event type or version and migration plan.

17. Staged evolution of the order platform

Stage 0: establish evidence and module boundaries

The initial eight-person team deploys one application. It creates Catalog, Ordering, Payments, Inventory, Fulfillment, and Notification modules, each with a public API and owned schema. A single relational database supplies local transactions. Module events run through an in-process dispatcher after the transaction or through an outbox when delivery must survive a crash. Tests forbid internal imports and cross-owner writes. Dashboards break down latency and errors by module.

Stage 1: separate asynchronous workers

Image work and notification delivery consume most campaign CPU and have weak consistency needs. Move them to queue workers while retaining their module ownership. The web deployment no longer scales for email bursts. Persist job state, deduplicate messages, bound retries, and reconcile stuck work. This solves asymmetric resource use without yet extracting core services.

Stage 2: extract payments for security and ownership

Payment handling now has a specialist team, restricted data, independent audit obligations, and a vendor release cycle. These are concrete reasons to extract it. Define payment intent, authorization, capture, refund, and idempotency contracts. Give the service its own credentials and data. Ordering stores payment references and status, not sensitive instrument data.

1. Client -PlaceOrder(request_id)-> Ordering
2. Ordering commits order=PENDING_PAYMENT and OrderPlaced outbox record
3. Workflow -Authorize(payment_id, amount, idempotency_key)-> Payments
4a. authorized -> Ordering records payment reference and continues reservation
4b. declined   -> Ordering marks order PAYMENT_FAILED
4c. timeout    -> Ordering queries by idempotency key before retrying
5. Reconciler resolves orders stuck beyond the state deadline

No database transaction spans Ordering and Payments.
Ambiguous outcomes are explicit states, not guessed failures.

Stage 3: extract inventory for scale and contention

Inventory traffic and locking now dominate campaigns, and a dedicated team owns replenishment and warehouse allocation. Create a reservation contract with expiry, confirmation, and release. The service owns stock quantities. Ordering owns the business order and its workflow. A saga tracks payment and reservation state. Reconciliation compares both authorities rather than sharing rows.

Stage 4: selective independence, not automatic extraction

Catalog may become independently deployed because read scale and release rate differ. Fulfillment may remain a module if one team and one release continue to work well. Notification can remain a worker. Architecture is now a hybrid. The success condition is lower lead time and controlled failure, not whether every box has become a service.

18. Strangler migration and anti-corruption layers

A strangler migration places a routing seam around a legacy capability, implements selected behavior in a new component, gradually moves traffic and data, and removes the old path only after evidence. It reduces big-bang risk and lets product work continue. Choose a vertical business slice, not only a shared utility, so ownership actually moves.

DISCOVER dependencies and invariants
   -> SHADOW new reads, compare without serving
   -> CANARY selected tenants or operations
   -> PRIMARY new owner, retain controlled rollback
   -> VERIFY correctness, SLO, cost, and recovery
   -> RETIRE legacy writer, schema, route, and adapter

Any unsafe mismatch -> stop traffic shift, reconcile, and return to last safe state

Anti-corruption layer

An anti-corruption layer translates a legacy or external model into the new bounded context's language. For example, a legacy integer payment_flag might map to explicit Payment states. Keep the translation beside the boundary and test it with historical edge cases. Do not allow the legacy model to spread inward. Give the adapter an owner and removal criterion, or it becomes permanent opaque middleware.

Data migration mechanics

1. Name one authoritative writer for each migration phase.
2. Backfill immutable snapshots with checkpoints and repeatable jobs.
3. Capture changes after the checkpoint through an outbox or change stream.
4. Compare counts, hashes, invariants, and sampled business results.
5. Switch reads gradually while monitoring stale and missing records.
6. Switch writes once, preserve idempotency, and rehearse rollback semantics.
7. Retire old credentials, jobs, routes, tables, and telemetry after the support window.

19. Distributed-monolith symptoms

A distributed monolith pays network and operational costs while retaining monolithic coupling.

How a distributed monolith fails

Catalog p99 150 ms
  -> Pricing p99 300 ms
      -> Customer p99 250 ms
          -> Inventory p99 600 ms
              -> Payment timeout 1,000 ms

Checkout timeout: 2,000 ms
Each layer retries twice with no shared deadline.
One user request can create up to 3 x 3 x 3 x 3 x 3 downstream attempts.
Queues and connection pools fill; health checks fail; replicas churn.

Pass one end-to-end deadline, retry at one controlled layer only when safe, budget attempts, bound concurrency, and shed excess load. More fundamentally, remove remote boundaries that merely split one cohesive request. An architecture problem cannot always be repaired by a retry library.

20. Data consistency and workflow ownership

Each service performs local transactions on owned data. Cross-service work becomes an explicit state machine. For an order, states might be PENDING_PAYMENT, PENDING_INVENTORY, CONFIRMED, REJECTED, or REVIEW_REQUIRED. Persist the next action, attempt identity, deadline, and last observed result. Use an outbox for reliable commands or events, inbox deduplication for consumers, and reconciliation for states that exceed their deadline.

21. Production deployment and operations

Minimum production service contract

• Named owner, repository, dependency inventory, data classification, and support channel.
• Versioned artifact, reproducible build, signed provenance, and dependency scanning.
• Configuration and secrets separated from the artifact with rotation procedures.
• Resource requests, limits, autoscaling signal, capacity model, and failure headroom.
• Readiness, liveness, startup, graceful draining, and bounded shutdown behavior.
• Compatibility-safe rollout, rollback limits, database migration strategy, and feature controls.
• SLIs, SLO, alerts, dashboard, traces, runbook, backup, restore, and recovery objective.

Independent deployment requires mixed versions to coexist. Deploy database expand changes before code that uses them, then contract only after all readers have migrated. Drain instances before stopping, stop accepting new work, finish or checkpoint bounded work, release leases, flush outbox state, and close connections within the termination budget. A liveness probe should detect a stuck process, not restart an instance merely because a downstream dependency is unavailable.

Governance without central paralysis

Set a small set of mandatory guardrails: supported identity and encryption, telemetry envelope, vulnerability policy, contract registry, data classification, ownership metadata, and deployment evidence. Offer approved templates and platform APIs. Let domain teams choose local internals within those limits. Review cross-cutting exceptions through short architecture decision records with owner, reason, consequences, expiry, and measurements.

Service catalog questions

• Who owns this service now, and who is paged?
• What business capability, contracts, data, and critical dependencies does it own?
• What are its SLO, recovery objectives, regions, and capacity limits?
• Which consumers and event schemas exist, and which versions are deprecated?
• Where are dashboard, runbook, deployment history, threat model, and decision records?

22. Observability and troubleshooting

Observe user journeys and boundaries, not just host health. Carry a trace context and stable business correlation identifier through synchronous calls and message envelopes. Never use the correlation identifier as an authorization decision. Structured logs should include service, version, environment, operation, outcome, latency, trace ID, and safe domain identifiers. Redact tokens, payment data, secrets, and unnecessary personal data.

Troubleshooting a slow checkout

1. Confirm the user-facing SLI, affected operations, tenants, regions, and versions.
2. Follow traces from edge to dependencies and compare latency budget with actual critical path.
3. Inspect saturation, pool waits, queue age, timeout, retry, and cancellation metrics.
4. Check recent deployment, schema, route, mesh, certificate, and feature changes.
5. Reduce load or disable noncritical work before adding retries or replicas blindly.
6. Verify workflow correctness and reconcile ambiguous attempts during recovery.
7. Record whether the boundary caused repeated operational coupling and revisit the design.

23. Security, privacy, abuse, and trust boundaries

Every process, service, queue, database account, gateway, sidecar, administrator, and external integration is a trust boundary with an identity and limited authority.

Identity and authorization

Authenticate users at the edge, but authorize sensitive business actions again at the service that owns the resource. Authenticate workloads with short-lived identities where possible and encrypt service traffic. Propagate only the user claims required for the operation, with audience and expiry checks. A service using its own broad identity on behalf of a user can become a confused deputy, so record actor, workload, tenant, purpose, and decision in the audit trail.

Least privilege by boundary

• Ordering can request an authorization but cannot read payment instrument secrets.
• Notification receives a purpose-specific delivery address, not an entire customer record.
• Analytics consumes minimized events and cannot call transactional write operations.
• Each service account can access only owned schemas, topics, secrets, and object prefixes.
• Administrative actions require stronger identity, reason capture, and immutable audit.

Abuse and input boundaries

Validate size, depth, count, encoding, and semantic constraints at every externally controlled contract. Rate-limit by trustworthy identity and costly operation, not IP alone. Bound fan-out, queue retention, decompression, file conversion, and downstream concurrency. Sign or authenticate webhooks, reject replay with timestamp and unique delivery ID, and make handlers idempotent. Treat broker messages as untrusted input because producer compromise and stale replay are possible.

Privacy and retention

Service-owned copies make access fast but multiply deletion and retention obligations. Maintain a data inventory and lawful purpose for each projection. Prefer opaque identifiers and minimized event payloads. Encrypt transit and storage, rotate keys, constrain backups, and test deletion propagation. A replay archive must not resurrect data after a valid deletion; use tombstones, cryptographic erasure where suitable, or rebuild from a policy-compliant source.

24. Performance, capacity, scalability, and cost

Latency and availability compose

Sequential calls add latency. Parallel calls take roughly the slowest branch plus coordination. Tail values do not combine by simply adding advertised p99 values, but every dependency adds opportunities for a slow result. If a request requires five services, each at 99.9 percent availability and failures were independent, the upper estimate is 0.999^5 = 99.501 percent. Shared infrastructure and correlated failures often make the real result worse. Cache or defer noncritical data, reduce fan-out, and assign hop deadlines from one end-to-end budget.

Capacity by resource and boundary

campaign checkout arrival = 120 requests/second
average checkout service time = 0.25 second
in-flight checkout work ~= 120 x 0.25 = 30 requests

inventory calls per checkout = 1
retry budget = at most 5 percent additional attempts
planned inventory arrival ~= 126 requests/second

If a worker processes 20 notifications/second and peak enqueue is 100/second,
five workers only keep pace. Add failure headroom and scale from oldest-message age.

Model CPU, memory, database connections, locks, network, payload size, storage growth, broker partitions, vendor quotas, and failure capacity. Independent scaling pays off only if the service owns the constrained resource. Splitting stateless code while all services contend for one database does not remove the bottleneck.

Cost model

• Baseline compute and replicas required for failure tolerance for every service.
• Load balancers, gateways, proxies, brokers, databases, telemetry, backups, and data transfer.
• Build pipelines, environments, contract infrastructure, patching, and incident response.
• Developer cognitive load, local setup, cross-team coordination, and slower debugging.
• Opportunity benefit from faster independent delivery, fault containment, or right-sized scaling.

A modular monolith often has lower fixed cost and more efficient local calls. Microservices can lower marginal coordination or scaling cost once independent capabilities and teams are real. Calculate total ownership cost and the value of the specific autonomy gained.

25. Testing boundaries, failure, and recovery

Test the smallest unit that can prove a property, then add boundary tests for the risks introduced by integration and distribution.

Use test doubles at owned contracts

A fake should implement a documented contract, including errors, latency, pagination, and idempotency, not expose provider internals. Keep a small suite against the real provider. Record production examples safely and add them as compatibility fixtures. Shared environments are useful for a few integration risks, but they must not be the only way a developer learns whether a change is safe.

Local development

• Run one service or module with owned database and broker substitutes.
• Provide versioned contract fixtures and deterministic seed data.
• Use a remote development environment only for dependencies impossible to reproduce safely.
• Make trace and message inspection available locally.
• Provide one command for build, unit, integration, and contract checks.
• Test mixed versions because production always contains version skew during rollout.

Fault and recovery campaign

Hypothesis:
  If Payments loses 20 percent of responses for 10 minutes,
  Ordering stays available for reads, creates no duplicate authorization,
  and reconciles 99 percent of pending attempts within 15 minutes.

Controls:
  bounded tenant cohort, abort threshold, named commander, rollback ready

Observe:
  pending-state age, idempotency conflicts, retries, payment count,
  checkout SLI, queue age, saturation, and reconciliation completion

After:
  verify business records, remove injection, document gaps, own follow-ups

26. Failure scenarios, edge cases, and corrections

Common design mistakes

• Premature split: prove modular boundaries and evidence first.
• Service per noun: group rules and outcomes by capability.
• Shared canonical domain model: translate between bounded contexts.
• Network inside a transaction: avoid holding locks across remote calls.
• Async by default: choose interaction from the user promise and failure model.
• Business rules in gateway or mesh: keep domain policy with its owner.
• Event notification without source data: define how consumers obtain a stable view.
• Infinite backward compatibility: publish support windows and remove with evidence.
• Ignoring deletion and replay: make privacy lifecycle part of event design.
• No merge strategy: accept that two wrongly split services may need to become one.

27. Architecture decision method

1. Clarify workload: users, operations, scale, latency, availability, privacy, cost.
2. Model domain: commands, events, invariants, aggregates, contexts, external systems.
3. Start simple: propose capability modules and one deployment unless evidence differs.
4. Trace critical paths: mark synchronous hops, transactions, trust, and failure states.
5. Evaluate forces: change coupling, ownership, scale, reliability, security, and cost.
6. Choose styles per problem: internal dependency rule, deployment, interaction, compute.
7. Design contracts: meaning, compatibility, idempotency, deadline, ownership, support.
8. Design failure first: ambiguous outcomes, replay, overload, migration, recovery.
9. Prove operations: SLO, capacity, telemetry, deploy, rollback, restore, and on-call.
10. Record decision: assumptions, alternatives, consequences, fitness functions, review date.

28. Hands-on exercises and design scenarios

Exercise 1: find module boundaries

Given catalog browsing, dynamic pricing, checkout, payment, warehouse stock, shipment, email, and analytics, draw commands, facts, invariants, and owners. Propose modules before services.

Exercise 2: challenge a microservice proposal

A team proposes Customer, Address, Order, OrderLine, Price, and Discount services because each is a database table. Review the checkout flow and redesign it.

Exercise 3: extract Payments

Design the API, data owner, workflow states, idempotency, timeout recovery, audit, and rollout from the modular monolith. State a rollback condition.

Exercise 4: choose sync or async

Decide how checkout obtains a price, reserves inventory, sends email, and feeds analytics.

Exercise 5: recover a distributed monolith

Five services share one database, deploy together, and form a synchronous chain. Create a three-month correction plan without a big-bang rewrite.

29. Interview questions and model answers

1. Is a monolith less scalable than microservices?

Not inherently. A stateless monolith can scale horizontally across many instances and use caches, queues, and database scaling. Its limitation is granularity: all capabilities share an artifact and usually a resource envelope. Microservices help when capabilities need meaningfully different scaling, isolation, ownership, or release cadence. They can scale worse if remote chattiness and shared databases dominate. A strong answer asks what resource is constrained and whether the split owns it. Follow-up: How would you isolate one CPU-heavy capability without extracting the whole domain? Move it to a bounded worker process and queue first.

2. How do you choose a service boundary?

Combine business language, invariants, aggregates, capability ownership, code change history, team boundaries, scaling shape, failure isolation, and data classification. Model end-to-end success and failure flows, then challenge the service as an in-process module. The boundary is credible when it is cohesive internally, has a small contract externally, owns its data, and has one operational owner. Follow-up: Is one bounded context always one service? No. It can remain a module or be deployed as more than one runtime for workload reasons.

3. What makes a modular monolith different from a traditional monolith?

The deployment count may be identical. The difference is enforced capability boundaries: explicit public APIs, hidden internals, owned tables, acyclic dependencies, module-level telemetry, and architecture tests. A traditional layered monolith often allows any service or repository to reach any model. Follow-up: Can a modular monolith use a queue? Yes. Deployment style and interaction style are independent.

4. Why should a microservice own its database?

Independent schema evolution and invariant enforcement require one writer and supported access paths. Shared tables create hidden contracts, security bypass, resource coupling, and coordinated releases. Database per service is logical ownership, not necessarily one physical server. Other services use APIs, events, or projections. Follow-up: How do you query across services? Compose APIs for low-volume current data or maintain an event-fed read model for frequent queries.

5. When should services communicate asynchronously?

Use asynchronous communication when the caller can expose pending state, work is deferrable, buffering protects a consumer, or several consumers react independently. It brings duplicates, ordering, replay, lag, and eventual consistency. Use synchronous communication when an immediate answer is necessary for the user promise. Follow-up: Does a broker remove coupling? No. It reduces temporal and location coupling while retaining semantic, schema, volume, and operational coupling.

6. How do you know services are independently deployable?

Verify that a provider can release with old and new consumers running, owns its migrations, has no shared release gate, passes compatibility checks, and can roll back within its data constraints. Measure independent deploy rate and coordinated change frequency. Separate repositories or pipelines do not prove it. Follow-up: What enables safe evolution? Additive contracts, expand-migrate-contract changes, support windows, consumer verification, and traffic evidence.

7. What problem does a service mesh solve?

A mesh standardizes service transport concerns such as workload identity, mTLS, discovery, balancing, traffic policy, and telemetry across a fleet. It does not solve service boundaries, business authorization, idempotency, or distributed transactions. It also adds proxy and control-plane cost and failure modes. Follow-up: When would you avoid it? A small, single-language fleet with adequate libraries may not repay the operational complexity.

8. Explain a safe strangler migration.

Put a route seam around one vertical capability, establish one authoritative writer, translate models through an anti-corruption layer, backfill and capture changes, shadow and compare, canary, then shift traffic. Reconcile continuously and keep a tested rollback that does not create two truths. Retire old routes, jobs, credentials, and data only after evidence. Follow-up: Why avoid a big bang? It delays feedback and combines domain, data, contract, and operational risk.

9. What are signs of a distributed monolith?

Lockstep deployment, shared tables, chatty synchronous chains, entity CRUD services, shared domain libraries, central business orchestration, and no independent ownership are strong signs. The system pays network failure costs but cannot change independently. Follow-up: Is merging services a failure? No. It can restore cohesion and remove unjustified remote boundaries.

10. Which architecture style would you choose for a new order system?

State assumptions first. For a small team and uncertain domain, choose a capability-oriented modular monolith, hexagonal boundaries for important adapters, and a queue worker for deferrable notifications. Enforce data ownership and instrument modules. Extract only where team autonomy, security isolation, scale, or blast radius becomes measurable. Follow-up: What would trigger payment extraction? Specialist ownership, restricted data, audit isolation, vendor release cadence, and an explicit workflow ready for ambiguous outcomes.

30. Revision cheat sheet

• Architecture style defines constraints; technology and deployment do not define the style.
• Layers separate technical responsibilities; modules separate business capabilities.
• Hexagonal and clean architecture point dependencies inward toward business policy.
• A modular monolith has one deployment but enforced APIs, data owners, and dependency rules.
• Microservices require independent deployment, data ownership, team ownership, and failure handling.
• Distribution is justified by measured autonomy, scale, isolation, or security needs.
• High cohesion means related rules and change belong together.
• Coupling can be source, contract, temporal, data, behavioral, operational, or organizational.
• A bounded context defines consistent language; an aggregate defines immediate consistency.
• Decompose by business capability, not table, screen, or technical layer.
• Database per service is logical ownership and one supported writer.
• Shared schemas create hidden contracts, security bypass, and coordinated releases.
• Synchronous calls need deadlines, cancellation, idempotency, and bounded retry.
• Async messaging needs durable acceptance, deduplication, ordering scope, replay, and recovery.
• Gateways own edge policy; BFFs adapt client needs; domain services own business rules.
• A mesh standardizes transport mechanics but cannot infer business correctness.
• Independent deployment requires compatible contracts and mixed-version operation.
• Prefer additive evolution and expand-migrate-contract removal.
• Consumer-driven tests complement, but do not replace, authoritative provider contracts.
• Strangler migration moves one vertical slice through shadow, canary, verify, and retire.
• An anti-corruption layer translates models and must have an owner and exit criterion.
• Avoid uncontrolled dual writes; use one authority plus durable change propagation.
• Lockstep services with shared data and chatty calls form a distributed monolith.
• Cross-service invariants need explicit workflow state, outbox, idempotency, and reconciliation.
• Observe user journeys, service boundaries, queue age, saturation, and stuck business states.
• Authorize at the data-owning service and minimize propagated identity and personal data.
• Model total cost, including baseline replicas, telemetry, environments, and cognitive load.
• Test domain, architecture, adapters, contracts, load, faults, replay, restore, and recovery.
• It is valid to keep a module, extract a service, or merge a wrong split based on evidence.

31. Primary official references

• Microsoft Azure Architecture Center: Architecture styles - definitions, constraints, and trade-offs for N-tier, web-queue-worker, microservices, and event-driven styles.
• Microsoft Azure Architecture Center: Microservices architecture style - capability decomposition, independent deployment, data ownership, gateways, and operational challenges.
• Microsoft Azure Architecture Center: Web-Queue-Worker architecture style - front-end and worker separation, queue decoupling, scaling, and dual-write considerations.
• Microsoft Azure Architecture Center: Backends for Frontends pattern - client-specific API adaptation, ownership, performance, and operational trade-offs.
• AWS Prescriptive Guidance: Strangler fig pattern - incremental migration, routing seams, applicability, and migration risks.
• AWS Prescriptive Guidance: Anti-corruption layer pattern - translating semantics between bounded contexts during migration and integration.
• IETF RFC 9110: HTTP Semantics - stateless request-response semantics, intermediaries, methods, status codes, and idempotency.
• OpenAPI Initiative: OpenAPI Specification - machine-readable HTTP API descriptions and published specification versions.
• CNCF CloudEvents Specification - interoperable event context attributes, formats, protocol bindings, and current stable release.
• Kubernetes documentation: Service - stable logical endpoints, EndpointSlices, DNS discovery, and changing backend instances.
• Kubernetes documentation: Sidecar containers - companion containers, lifecycle, startup, and shutdown behavior.
• Envoy official documentation: Life of a Request - proxy request flow and service-mesh transport responsibilities.
• Envoy official documentation: Service discovery - endpoint discovery, health checking, eventual consistency, and routing behavior.
• PostgreSQL documentation: Privileges - database, schema, table, sequence, and function permissions used to enforce ownership boundaries.