System Design Interview Tutorial · Chapter 5

Architectural patterns in depth

The overview chapter maps the patterns and helps you choose between them. This chapter is the reference for its high-level architecture level. For each pattern you get:

Don't read it front to back. Jump to the pattern that comes up in your practice question, and use the numbers to back a trade-off out loud. The lower levels have their own chapters: low-level primitives (transactions, logs, consensus) and infrastructure building blocks (queues, outbox, caches, gateways).

Keep the overview's rule in mind throughout: decide who owns the authoritative state and which guarantee you need, then pick the smallest pattern that keeps that guarantee under the failures you must tolerate. CRUD, CQRS, event sourcing and the outbox are separate decisions; the sections below keep them separate.

Application structure

These patterns decide how code and data are organized, inside one process or across services.

CRUD and transactional application

What it is. An application that stores current state as rows and changes it with create, read, update and delete operations inside local database transactions. It is the right default for most systems.

How it works. A stateless handler validates a request, opens a transaction, reads and writes rows, and commits. The database enforces invariants with constraints (unique keys, foreign keys, CHECK), and concurrency control protects read-modify-write sequences. Two safe ways to update a balance:

-- Conditional update: the database checks the invariant atomically
UPDATE accounts SET balance = balance - 30
WHERE id = 42 AND balance >= 30;          -- 0 rows updated = insufficient funds

-- Optimistic concurrency: detect a concurrent writer with a version column
UPDATE orders SET status = 'CANCELLED', version = version + 1
WHERE id = 7 AND version = 12;            -- 0 rows updated = retry or reject

CRUD describes the storage model, not the API. A domain command such as CancelOrder is still CRUD underneath.

flowchart LR
  C[Client] --> LB[Load balancer]
  LB --> A1["App instance (stateless)"]
  LB --> A2["App instance (stateless)"]
  A1 --> DB[("Primary database")]
  A2 --> DB
  DB -.->|async replication| R[("Read replica")]
  A1 -.->|stale-tolerant reads| R

Numbers. Optimistic concurrency fails when another write lands inside your read-to-write window. If a row receives λ updates per second and each transaction holds the version for T seconds, the chance of a conflict is roughly 1 − e^(−λT). At λ = 5/s and T = 50 ms, λT = 0.25, so about 22% of attempts retry. At 0.1 updates/s the rate falls to 0.5%. That is why optimistic locking suits rarely contended rows, and why a hot row (a global counter, a popular item's stock) needs a conditional update, a lock or sharding of the counter.

A single well-tuned relational primary handles roughly thousands to tens of thousands of simple write transactions per second, depending on hardware, row size and durability settings. Stack Overflow served one of the busiest sites on the web from a handful of SQL Server machines for years.

Where it's used. Most SaaS products, admin tools, catalogs, CRMs and back offices. Stack Overflow's 2016 architecture is a well-known example of a large CRUD system on a few databases.

Trade-offs and failure modes.

Sources: PostgreSQL transaction isolation, Fowler, Optimistic Offline Lock.

Transaction script, layered design and domain model

What it is. Three ways to organize business logic. A transaction script is one procedure per use case. A layered design separates presentation, application, domain and data access. A domain model puts rules on objects that guard their own invariants, grouped into aggregates (Domain-Driven Design).

How it works. A transaction script reads rows, applies rules inline and writes rows. It is simple and fast until rules interact. A domain model loads an aggregate, calls a method that checks invariants, and saves the aggregate in one transaction. An aggregate is the consistency boundary: everything inside it is consistent at commit; anything outside is updated by reference or eventually.

class Order (aggregate root)
  lines: List<OrderLine>
  status: Draft | Placed | Cancelled
  version: int

  addLine(sku, qty, price):
    require status == Draft
    require lines.count < 100            -- invariant lives here
    lines.add(OrderLine(sku, qty, price))

  place():
    require lines not empty
    status = Placed
    record OrderPlaced(id, total())      -- optional domain event

Vaughn Vernon's rules for aggregates: protect true invariants inside one boundary, keep aggregates small, reference other aggregates by ID, and update other aggregates eventually.

flowchart TB
  P["Presentation (HTTP, CLI)"] --> AS["Application service (use case, transaction)"]
  AS --> D["Domain model (aggregates, invariants)"]
  AS --> R["Repository interface"]
  R --> I["Infrastructure (SQL, ORM)"]
  I --> DB[("Database")]

Numbers. Aggregate size drives contention. Using the conflict formula from the CRUD section, an aggregate edited by u users each writing at rate r has λ = u × r. A "Project" aggregate holding all tasks, with 20 users each saving every 30 s, has λ ≈ 0.67/s; with T = 100 ms about 6% of saves conflict. Split tasks into their own aggregates and each sees far less. Hidden lazy loading costs too: listing 100 orders that each lazily load lines issues 1 + 100 queries; at 0.5 ms per round trip that is about 50 ms against roughly 1 ms for one join.

Approach Use when Breaks down when
Transaction script Few, independent rules; CRUD screens Rules duplicate across scripts
Layered design Most applications; clear separation of concerns Layers only pass data through
Domain model Interacting invariants, rich lifecycle Simple data entry; heavy reporting queries

Where it's used. Fowler's Patterns of Enterprise Application Architecture named transaction script and domain model. DDD aggregates are standard in enterprise Java and .NET codebases, and they map directly to the event-sourced aggregates and actors later in this chapter.

Trade-offs and failure modes. Too many layers obscure simple work. Large aggregates cause lock contention and slow loads. An invariant that spans aggregates cannot be enforced in one transaction; either redraw the boundary or accept eventual correction with a process manager.

Sources: Fowler, Domain Model, Fowler, Transaction Script, Vernon, Effective Aggregate Design.

Modular monolith

What it is. One deployable application split into modules along business boundaries, where each module exposes a public interface, owns its data and may depend only on declared modules.

How it works. Code moves from technical folders (models/, controllers/) into domain folders (orders/, billing/, inventory/). Each module has:

Modules still call each other in process and can share one database transaction when they must. That keeps strict invariants cheap while the boundaries settle.

flowchart LR
  subgraph App["One deployable"]
    O["Orders module"] -->|public API| I["Inventory module"]
    O -->|public API| B["Billing module"]
    B -->|public API| C["Customers module"]
  end
  O --> S1[("orders schema")]
  I --> S2[("inventory schema")]
  B --> S3[("billing schema")]
  C --> S4[("customers schema")]

Numbers. An in-process call costs nanoseconds to microseconds; a call across a network in one data center costs roughly 0.5–1 ms plus serialization, and adds a failure mode. A request that crosses 6 module boundaries pays nothing extra in a monolith and 3–6 ms plus 6 failure points as microservices. Shopify reported that its core monolith had about 6,000 Ruby classes, manually assigned to components, worked on by more than a thousand developers over a decade. Its Packwerk tool later enforced 48 packages in the core monolith.

Where it's used. Shopify's core Rails application is the best-documented example. Shopify built Wedge to score each component's isolation from call graphs collected in CI, and then Packwerk, which statically checks package.yml dependency and privacy rules. Fowler's "Monolith First" argues that most successful microservice systems started this way.

Trade-offs and failure modes.

Sources: Shopify, Deconstructing the Monolith, Shopify, Packwerk, Fowler, Monolith First.

Bounded-context services and private data ownership

What it is. Independently deployable services, each aligned to a bounded context (a part of the domain with its own consistent language and model) and each the only owner of its data: database per service.

How it works. A service owns its tables; no other service reads or writes them. Others get data through the owner's API or through events it publishes. "Database per service" can mean a private schema, a private database on a shared server, or a separate server; the rule is about access, not hardware.

Conway's law shapes the boundaries: "organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations." Teams that own one context end to end can release without coordination. Cutting services that cross team lines produces chatty interfaces and coordinated deploys.

flowchart LR
  subgraph TA["Team A"]
    OS["Order service"] --> ODB[("orders DB")]
  end
  subgraph TB["Team B"]
    PS["Payment service"] --> PDB[("payments DB")]
  end
  subgraph TC["Team C"]
    SS["Shipping service"] --> SDB[("shipping DB")]
  end
  OS -->|API call| PS
  OS -.->|OrderPlaced event| SS

Numbers. Communication paths in a team grow as n(n − 1)/2: 8 people have 28 paths, 20 people have 190. Small teams that own one service each keep that number low; that is the reasoning behind Amazon's two-pizza teams. The cost appears in availability and latency. A request that synchronously needs k services, each 99.9% available, is at most 0.999^k available: 99.5% for k = 5, 99.0% for k = 10. Cross-service queries become network joins: fetching 50 orders and then each customer one by one is 51 round trips; batch the lookup.

Where it's used. Amazon moved from a two-tier monolith to services that own their data; Werner Vogels described it in a 2006 ACM Queue interview, including the rule that service data is accessible only through the service's interface ("you build it, you run it" comes from the same interview). The microservices.io database-per-service pattern describes the data rule.

Trade-offs and failure modes.

Sources: Conway, How Do Committees Invent? (1968), Vogels interview, ACM Queue 2006, Fowler, Bounded Context.

Shared database across services

What it is. Several services read and write the same database tables. It is fine as a deliberate transition step and harmful as a permanent design.

How it works. Each service connects to one schema and runs its own queries. Cross-service invariants can use ordinary transactions and joins, which is why it is tempting. The coupling is invisible: a column rename or a long-running query in one service breaks or slows another.

flowchart TB
  A["Order service"] --> DB[("Shared database")]
  B["Billing service"] --> DB
  C["Reporting service"] --> DB
  DB --- T1["orders table"]
  DB --- T2["invoices table"]
  C -.->|reads both, blocks schema changes| T1

Numbers. Two costs grow with the number of services S:

Where it's used. Most monolith-to-services migrations pass through this stage. Chris Richardson documents it as the "shared database" pattern, with its drawbacks next to database per service.

Trade-offs and failure modes.

Sources: microservices.io, Shared database, PostgreSQL connection settings.

Synchronous API composition and backends for frontends

What it is. API composition answers a query by calling several services and merging their responses. A backend for frontend (BFF) is a composition layer built for one client type (iOS, web, TV) and usually owned by that client's team.

How it works. The composer receives a client request, fans out calls (in parallel when they don't depend on each other), applies per-call deadlines, merges, and trims the payload for the client. A BFF adds client-specific shaping: fewer fields for a TV, a different page structure for mobile. Gateways handle cross-cutting concerns (auth, rate limits, routing); BFFs handle client-specific composition; business rules stay in the services.

sequenceDiagram
  participant M as Mobile app
  participant B as Mobile BFF
  participant P as Profile service
  participant O as Order service
  participant R as Recommendation service
  M->>B: GET /home
  par Parallel calls with 150 ms deadline
    B->>P: GET /profile/42
    B->>O: GET /orders?user=42&limit=3
    B->>R: GET /recs/42
  end
  P-->>B: profile
  O-->>B: 3 orders
  R--xB: timeout
  B-->>M: home screen without recommendations

Numbers.

Where it's used. Netflix redesigned its API around device-specific endpoints: client teams wrote Groovy adapter scripts on the server that fanned out to Java service APIs, to reduce chattiness and payload size. Sam Newman named the BFF pattern from work at SoundCloud. The API gateway itself is covered in the infrastructure chapter.

Trade-offs and failure modes.

Sources: Newman, Backends For Frontends, microservices.io, API composition, InfoQ, Netflix API optimization.

Hexagonal architecture (ports and adapters)

What it is. A way to structure one application so the domain core depends only on interfaces it defines (ports), and every technology (HTTP, database, message broker, payment provider) plugs in through adapters. Alistair Cockburn named it in 2005.

How it works. Dependencies point inward. Driving (primary) adapters call the application: an HTTP controller, a queue consumer, a test. Driven (secondary) adapters implement ports the core needs: a repository, a payment gateway, a clock.

// Driven port, defined by the core in the core's language
interface PaymentGateway {
  authorize(orderId: OrderId, amount: Money, idempotencyKey: string): AuthResult
}

// Driving port: the use case the core offers
interface PlaceOrder {
  execute(cmd: PlaceOrderCommand): OrderId
}

// Adapters live outside the core
class StripePaymentGateway implements PaymentGateway { ... }   // production
class FakePaymentGateway  implements PaymentGateway { ... }    // tests

flowchart LR
  HTTP["HTTP controller"] --> IN["Driving port: PlaceOrder"]
  MQ["Queue consumer"] --> IN
  IN --> CORE["Domain core"]
  CORE --> OUT1["Driven port: OrderRepository"]
  CORE --> OUT2["Driven port: PaymentGateway"]
  OUT1 --> PG["Postgres adapter"]
  OUT2 --> ST["Payment provider adapter"]
  OUT2 -.-> FK["Fake adapter for tests"]

Numbers. The main payoff is test speed and isolation. A domain test against in-memory adapters runs in roughly a millisecond; one that starts a database and calls a sandbox API takes roughly 100–500 ms. A suite of 2,000 domain tests runs in about 2 s instead of 3–15 minutes, which changes how often engineers run it.

Where it's used. Netflix described moving a studio application to hexagonal architecture so it could swap data sources (from a monolith's API to new services) without touching business logic. The pattern is close to Clean Architecture and Onion Architecture; all three keep the domain free of framework imports.

Trade-offs and failure modes.

Sources: Cockburn, Hexagonal Architecture, Netflix, Ready for changes with Hexagonal Architecture.

Data and interactions

These patterns decide how state changes travel between owners and how reads are served.

Event-driven integration

What it is. Services communicate by publishing facts about what happened ("OrderPlaced") to a broker, and other services react on their own schedule. The publisher doesn't know or wait for its consumers.

How it works. The producer commits a state change and, through a transactional outbox or a durable event store, publishes an event to a topic. Each consumer group reads at its own pace, dedupes by event ID, and updates its own data. Events come in two shapes:

Shape Payload Consumer does Trade-off
Event notification {"type":"OrderChanged","orderId":42} Calls back to the producer for details Small, stable schema; callback load and coupling at read time
Event-carried state transfer Full order snapshot or the changed fields Acts alone, keeps a local copy Larger payloads, stronger schema contract, consumers hold stale copies

Schema evolution. Treat events as a public API. Use a schema registry (Avro, Protobuf or JSON Schema) with a compatibility rule. Backward compatibility (new consumers can read old events) allows adding optional fields and removing fields; full compatibility also lets old consumers read new events. Never reuse a field's name with a new meaning; publish a new event type or version instead. Include eventId, eventType, version, occurredAt and the entity key in every envelope.

flowchart LR
  OS["Order service"] -->|commit + outbox| OB[("orders DB + outbox")]
  OB --> RL["Outbox relay or CDC"]
  RL --> T["Topic: orders, keyed by orderId"]
  T --> G1["Shipping consumer group"]
  T --> G2["Email consumer group"]
  T --> G3["Analytics consumer group"]
  SR["Schema registry"] -.-> RL
  SR -.-> G1

Numbers.

Where it's used. Kafka was built at LinkedIn to move activity and change events between systems; it is now the default backbone for event-driven integration. Confluent Schema Registry defaults to backward compatibility.

Trade-offs and failure modes.

Sources: Fowler, What do you mean by "Event-Driven"?, Confluent, Schema evolution and compatibility, Kafka design.

CQRS

What it is. Command Query Responsibility Segregation: one model handles writes (commands that enforce rules), a different model handles reads (queries shaped for screens). It does not require separate databases, events or event sourcing.

How it works. Two levels:

Level Write side Read side Freshness
Same database Normalized tables, domain model SQL views, materialized views or separate read tables updated in the same transaction Immediate (or at refresh for materialized views)
Separate read store Normalized tables or event store Elasticsearch, Redis, a denormalized table, a column store, fed by events or CDC Eventually consistent: projection lag

A projector consumes changes in order per entity, updates the read model idempotently, and stores its checkpoint (log offset or last event version) in the same transaction as the read-model update. Rebuilding means resetting the checkpoint and replaying.

flowchart LR
  U["Client"] -->|command| CH["Command handler"]
  CH --> W[("Write model")]
  W -->|change stream| PJ["Projector (stores checkpoint)"]
  PJ --> RV[("Read model")]
  U -->|query| QH["Query handler"]
  QH --> RV

Numbers.

Where it's used. Greg Young popularized the term; Microsoft's Azure Architecture Center and Fowler both describe it and warn it adds risk to most systems. A common real form: an order database for writes plus a search index or denormalized "order history" table for the customer's list page.

Trade-offs and failure modes.

Sources: Fowler, CQRS, Azure, CQRS pattern.

Event sourcing

What it is. The authoritative record of an entity is its ordered list of domain events; current state is derived by folding over them. A service is event-sourced only when that history governs its state changes, not when it merely publishes or archives events.

How it works. Each aggregate has a stream (order-42). To handle a command: load the latest snapshot if any, replay later events to rebuild state, decide, and append new events with the expected version. The store rejects the append if another writer got there first; that is optimistic concurrency on the stream.

handle(cmd: CancelOrder):
  (state, version) = load("order-" + cmd.orderId)   -- snapshot + later events
  if state.status == Shipped: reject("already shipped")
  events = [OrderCancelled(cmd.orderId, cmd.reason, now())]
  store.append("order-" + cmd.orderId, events, expectedVersion = version)
  -- WrongExpectedVersion: reload and retry, or reject

-- Minimal relational event store
CREATE TABLE events (
  stream_id   text,
  version     bigint,
  global_pos  bigserial,
  type        text,
  schema_ver  int,
  payload     jsonb,
  metadata    jsonb,
  PRIMARY KEY (stream_id, version)   -- the unique key enforces the expected-version check
);

Projections subscribe to the global order (global_pos) to build read models, as in CQRS. Snapshots store folded state every k events to bound load time. Versioning: old events are immutable, so an upcaster converts OrderPlaced v1 into v2 at read time (for example, adding a default currency). For corrections, append a compensating event; don't edit history.

Sensitive data and GDPR. Immutable history conflicts with the right to erasure. Options: keep personal data out of events and reference it by ID; or crypto-shredding: encrypt each person's fields with a per-person key held in a separate key store, and delete the key to make those fields unreadable everywhere, including backups.

flowchart LR
  CMD["Command"] --> AG["Aggregate: load, decide"]
  ES[("Event store, streams")] -->|snapshot + tail events| AG
  AG -->|append, expected version| ES
  SN[("Snapshots")] -.-> AG
  ES -->|ordered subscription| P1["Projection: order list"]
  ES -->|ordered subscription| P2["Projection: revenue"]
  ES -->|ordered subscription| PUB["Publisher to other services"]

Numbers.

Where it's used. Accounting ledgers are the oldest form: entries are never erased, only reversed. LMAX's exchange runs its business logic in memory with event sourcing, snapshots nightly, and restarts by replaying a day's journal in under a minute. EventStoreDB (now KurrentDB) and Marten (on PostgreSQL) are purpose-built stores; both expose expected-version appends.

Trade-offs and failure modes.

Sources: Fowler, Event Sourcing, Azure, Event Sourcing pattern, Fowler, The LMAX Architecture.

Business coordination

These patterns coordinate work that spans owners, services or time.

Saga orchestration

What it is. A long-running business transaction split into local transactions, driven by a central orchestrator that tells each participant what to do and runs compensating actions when a step fails. Garcia-Molina and Salem introduced sagas in 1987.

How it works. The orchestrator persists the saga's state before and after each step, so a crash resumes from the last recorded step. Each step is a command with an idempotency key (saga ID plus step name); each compensation is a business action (refund, release), not a rollback. Steps after a pivot (the point of no return, such as shipping) are retried until they succeed instead of compensated.

saga_instance
  saga_id        uuid PK
  type           'PlaceOrder'
  state          'PAYMENT_PENDING'
  data           jsonb      -- orderId, amount, reservationId ...
  version        int        -- optimistic lock on the instance
  deadline_at    timestamp  -- timer for the current step
  updated_at     timestamp

stateDiagram-v2
  [*] --> Pending
  Pending --> InventoryReserved: reserve ok
  Pending --> Rejected: out of stock
  InventoryReserved --> PaymentAuthorized: authorize ok
  InventoryReserved --> ReleasingInventory: payment declined or timeout
  ReleasingInventory --> Cancelled: released
  PaymentAuthorized --> Shipped: ship ok, pivot
  Shipped --> Completed: capture payment, retried until ok
  Rejected --> [*]
  Cancelled --> [*]
  Completed --> [*]

sequenceDiagram
  participant O as Orchestrator
  participant I as Inventory
  participant P as Payment
  O->>I: Reserve items (key saga-7:reserve)
  I-->>O: Reserved
  O->>P: Authorize 80 USD (key saga-7:pay)
  P-->>O: Declined
  Note over O: persist state ReleasingInventory
  O->>I: Release reservation (key saga-7:release)
  I-->>O: Released
  Note over O: persist state Cancelled, notify customer

Numbers. If step i fails with probability fᵢ, the share of sagas that compensate is 1 − Π(1 − fᵢ). Four steps at 1% each: 1 − 0.99^4 ≈ 3.9%. At 100 orders/s, about 4 sagas per second compensate, and compensations themselves fail sometimes, so plan a queue of sagas needing human repair. The orchestrator costs about 2 durable writes per step (command sent, reply recorded); a 4-step saga at 100/s needs about 800 writes/s plus timer storage.

Where it's used. Uber built Cadence for long-running business logic; its creators forked it as Temporal. AWS Step Functions and Azure Durable Functions provide the same model as managed services. In these engines, workflow code is deterministic and replayed from recorded history, while activities (the side-effecting calls) are retried and may run more than once.

Trade-offs and failure modes.

Sources: Garcia-Molina and Salem, Sagas (1987), microservices.io, Saga, Temporal, Workflow Execution, AWS, Saga orchestration.

Saga choreography

What it is. The same multi-step transaction without a coordinator: each participant listens for events, does its local transaction, and emits the next event. Compensation is also triggered by events.

How it works. Order publishes OrderCreated; Inventory reserves and publishes InventoryReserved; Payment charges and publishes PaymentCompleted or PaymentFailed; on PaymentFailed, Inventory releases and Order cancels. Each service persists its own part of the state and dedupes events by ID. Nobody holds the whole process; it exists only in the subscriptions.

sequenceDiagram
  participant O as Order
  participant B as Event broker
  participant I as Inventory
  participant P as Payment
  O->>B: OrderCreated
  B->>I: OrderCreated
  I->>B: InventoryReserved
  B->>P: InventoryReserved
  P->>B: PaymentFailed
  B->>I: PaymentFailed
  I->>B: InventoryReleased
  B->>O: PaymentFailed
  Note over O: mark order Cancelled

Orchestration vs choreography.

Concern Orchestration Choreography
Where the process lives One workflow definition Spread across subscribers
Timeouts ("payment never answered") Orchestrator timer Someone must own a timer; often nobody does
Adding a step Change the orchestrator Change subscriptions in several services
Coupling Orchestrator knows participants' commands Participants know each other's events
Messages per step Command + reply (2) One event (1), fanned out
Visibility Query the saga state Reconstruct from traces and logs
Best for Branching, timers, many steps, human steps 2–4 stable participants, simple linear flows

Numbers. Choreography's hidden cost is understanding, not messages. A flow with n participants where each may react to any other's events has up to n(n − 1) possible subscription edges: 12 for 4 services, 30 for 6. Each edge is a contract to evolve and a possible cycle. Message volume is lower: a 4-step choreographed saga emits about 4 events against about 8 messages for orchestration.

Where it's used. Common in event-driven systems built on Kafka or Amazon EventBridge where a few services react to domain events. AWS's prescriptive guidance describes it alongside orchestration and recommends it for simpler flows.

Trade-offs and failure modes.

Sources: AWS, Saga choreography, microservices.io, Saga.

Reservation and Try-Confirm-Cancel

What it is. Hold a scarce resource provisionally, then confirm or release it. Try-Confirm-Cancel (TCC) makes this a protocol: every participant offers try (reserve), confirm (make permanent) and cancel (release), and a reservation expires on its own if neither arrives.

How it works. try creates a reservation row with a TTL and decrements available stock in one local transaction. confirm turns it into a sale; cancel or expiry returns the stock. All three are idempotent on the reservation ID, and a late confirm after expiry must fail cleanly.

-- Try: reserve 2 seats if available, in one transaction
UPDATE inventory SET available = available - 2
WHERE sku = 'SEAT-A' AND available >= 2;
INSERT INTO reservations (id, sku, qty, status, expires_at)
VALUES ('r-91', 'SEAT-A', 2, 'HELD', now() + interval '10 minutes');

-- Confirm: only a HELD, unexpired reservation can be confirmed
UPDATE reservations SET status = 'CONFIRMED'
WHERE id = 'r-91' AND status = 'HELD' AND expires_at > now();

stateDiagram-v2
  [*] --> Held: try, stock decremented
  Held --> Confirmed: confirm before expiry
  Held --> Cancelled: cancel
  Held --> Expired: TTL elapsed, sweeper
  Cancelled --> [*]: stock returned
  Expired --> [*]: stock returned
  Confirmed --> [*]

Numbers. By Little's law, units held at any moment = reservation rate × hold time. At 200 reservations/s with a 10 minute TTL and an average hold of 4 minutes for completed checkouts, but the full 10 minutes for the 30% that abandon, the average hold is 0.7 × 240 + 0.3 × 600 = 348 s, so about 70,000 units are held. For a 50,000 seat event that sells out in holds before anyone pays. Shorter TTLs or a per-user hold cap fix it. Sweepers add lag: with a sweep every 30 s, stock returns up to 30 s late.

Where it's used. Ticketing seat holds, hotel and airline bookings, flash sales, and cloud capacity reservations. Atomikos described TCC for REST services, and Apache Seata (from Alibaba) offers a TCC mode. Pat Helland's "Life beyond Distributed Transactions" describes the same tentative-operation idea between entities.

Trade-offs and failure modes.

Sources: Helland, Life beyond Distributed Transactions, Apache Seata, TCC mode, microservices.io, Saga.

Scatter-gather

What it is. Send one request to many recipients in parallel and aggregate their answers under an explicit completion rule.

How it works. An aggregator splits the request (by shard, provider or region), sends all parts with a shared deadline, and merges results as they arrive. The completion rule decides when to answer:

Rule Answers when Example
All Every branch responds Exact count across shards
Deadline Time is up, with whatever arrived Search results, price quotes
Quorum or required subset Enough or the critical branches responded Replicated reads, mandatory providers
Good enough First k acceptable answers Cheapest of several suppliers

The response must say whether it is partial. Hedged requests (send a second copy after the p95 delay) reduce tails when there is spare capacity.

flowchart LR
  Q["Query"] --> AGG["Aggregator, deadline 100 ms"]
  AGG --> S1["Shard 1"]
  AGG --> S2["Shard 2"]
  AGG --> S3["Shard 3"]
  AGG --> S4["Shard 4"]
  S1 --> M["Merge top-k, mark partial"]
  S2 --> M
  S3 --> M
  S4 -.->|late, dropped| M
  M --> RES["Response"]

Numbers. If each branch is fast with probability p, all n are fast with probability pⁿ. "The Tail at Scale" gives the canonical example: if one server's p99 is 1 s and a request fans out to 100 servers, 63% of requests take more than 1 s (1 − 0.99^100 ≈ 0.63). Load multiplies too: 1,000 queries/s over 50 shards is 50,000 shard requests/s. A two-phase search (query then fetch) doubles the round trips.

Where it's used. Web search serving trees at Google (the source of "The Tail at Scale"). Elasticsearch runs every search as scatter-gather across shards, with query and fetch phases. Travel and insurance sites gather quotes from many providers under a deadline.

Trade-offs and failure modes.

Sources: Dean and Barroso, The Tail at Scale, AWS, Scatter-gather, Hohpe, Enterprise Integration Patterns: Scatter-Gather.

Pipes and filters

What it is. A processing job decomposed into independent stages (filters) connected by channels (pipes). Each stage does one transformation and can scale, deploy and fail on its own.

How it works. A stage reads from its input queue or topic, transforms, writes to its output, and commits its input position after the output is durable. That gives at-least-once processing, so each stage must be idempotent or deduplicate. Bounded channels apply backpressure; checkpoints let a restarted stage resume. Stream processors (Kafka Streams, Flink) add exactly-once state within their own boundary by checkpointing state and offsets together.

flowchart LR
  UP["Upload"] --> Q1[["queue: raw"]]
  Q1 --> F1["Validate x2"]
  F1 --> Q2[["queue: valid"]]
  Q2 --> F2["Transcode x20"]
  F2 --> Q3[["queue: encoded"]]
  Q3 --> F3["Thumbnail x4"]
  F3 --> Q4[["queue: done"]]
  Q4 --> F4["Publish metadata"]
  F1 -.->|poison| DLQ[["dead letters"]]

Numbers. Pipeline throughput is the slowest stage: X = min(instancesᵢ × rateᵢ). With validation at 2 × 500/s = 1,000/s, transcoding at 20 × 2/s = 40/s and thumbnails at 4 × 50/s = 200/s, the pipeline does 40 videos/s; the queue in front of transcoding grows at arrivals − 40 per second. End-to-end latency is Σ (queue wait + service time). Scale the bottleneck stage, not all stages.

Where it's used. Unix pipelines are the original. Media pipelines (upload, transcode, thumbnail), log ingestion, ETL, and stream processing in Kafka Streams and Apache Flink. Microsoft documents it as a cloud design pattern.

Trade-offs and failure modes.

Sources: Azure, Pipes and Filters, Flink, Fault tolerance via state snapshots.

Single-writer entity and actor ownership

What it is. Every entity (an account, a game session, a cart) has exactly one owner at a time that processes its commands sequentially. Owning the entity removes locks: the owner's in-memory state is the serialization point.

How it works. Commands for an entity are routed by its ID to the node that hosts it. The owner handles one message at a time from a mailbox, updates memory, and persists (a database write, an event append, or a journal). Frameworks differ in how ownership is placed:

Ownership transfer is the hard part: a paused old owner must not keep writing. Use leases with fencing, or make the persistence layer reject stale writers (an expected version on append).

flowchart LR
  C1["Command for cart-17"] --> RT["Router: hash(id) to shard"]
  C2["Command for cart-99"] --> RT
  RT --> N1["Node A: shards 1-50"]
  RT --> N2["Node B: shards 51-100"]
  N1 --> E1["Entity cart-17, mailbox, serial"]
  N2 --> E2["Entity cart-99, mailbox, serial"]
  E1 --> J[("Journal or database, version check")]
  E2 --> J
  CO["Shard coordinator"] -.->|assigns shards| N1
  CO -.-> N2

Numbers. One entity's throughput is capped at 1 ÷ service time: at 1 ms per command, 1,000 commands/s per entity, however many nodes you add. If each command waits on a 5 ms database write, the cap drops to about 200/s unless you batch writes. Across entities it scales linearly. Doing the work in memory on one thread can be very fast: LMAX reported 6 million orders per second on a single thread.

Where it's used. Orleans has run in production at Microsoft since 2011 and powered the Halo 4 services. Akka Cluster Sharding is used for carts, sessions and IoT devices. LMAX's exchange uses a single-writer business logic processor.

Trade-offs and failure modes.

Sources: Bernstein et al., Orleans: Distributed Virtual Actors, Akka Cluster Sharding, Fowler, The LMAX Architecture.

Evolution and isolation

These patterns decide how systems change safely and how far a failure can spread.

Anti-corruption layer

What it is. A translation layer between your domain and a legacy or external system, so the other model's concepts, codes and quirks never leak into yours. Eric Evans named it in Domain-Driven Design.

How it works. The ACL exposes an interface in your language (CustomerProfile, ShippingQuote) and implements it by calling the other system and translating: field mapping, code tables ("ST=07" to Shipped), unit and currency conversion, error mapping, and sometimes caching and retries. In hexagonal terms it is a driven adapter with real logic. It can be a class inside your service or its own service when several consumers need it.

flowchart LR
  subgraph New["Your bounded context"]
    D["Domain logic"] --> P["Port: CustomerDirectory"]
  end
  P --> ACL["Anti-corruption layer: map, translate, cache"]
  ACL --> L["Legacy CRM (SOAP, status codes)"]
  ACL --> V["Vendor API (different IDs)"]

Numbers. An ACL as a separate service adds one network hop, roughly 1 ms in the same data center, and its own availability term. A cache in the ACL can protect a fragile legacy system: if the legacy system handles 200 requests/s and the ACL serves 90% from cache, the new system can sustain about 2,000 requests/s (200 ÷ (1 − 0.9)), as long as the data can tolerate the cache TTL.

Where it's used. Any migration from a mainframe, ERP or CRM; integrating payment, shipping or identity vendors whose models differ from yours. AWS and Microsoft both document it as a modernization pattern.

Trade-offs and failure modes.

Sources: AWS, Anti-corruption layer, Azure, Anti-corruption Layer.

Strangler fig migration

What it is. Replace a legacy system incrementally: put a routing layer in front, move one capability at a time to the new system, and retire the old one when nothing routes to it. Martin Fowler named it after strangler figs he saw in Queensland in 2001.

How it works. Four moves, repeated per capability:

  1. Intercept: a proxy or gateway in front of the legacy system routes by path, header or tenant.
  2. Build and shadow: the new service receives a copy of real traffic; responses are compared but discarded.
  3. Move reads, then writes: shift traffic gradually (1%, 5%, 25%, 100%), then cut over write ownership: at any moment exactly one system is the authority for each record.
  4. Retire: remove the legacy path and its data.

Data during the transition. Two ways to keep both sides populated:

Approach How Risk
Dual writes from application code Write to old and new stores in one request No atomicity: a crash between writes leaves them divergent; needs reconciliation
CDC backfill Snapshot the legacy table, then stream its change log into the new store Snapshot and stream must line up (start the stream position before the snapshot); new side lags
Write-through ownership flip New system owns writes and syncs back to legacy via CDC or events Legacy consumers see lag; flip must be per entity or per tenant

Prefer CDC or outbox-based sync over application dual writes. Compare old and new continuously; a mismatch rate above a small threshold blocks the next step.

sequenceDiagram
  participant C as Client
  participant R as Routing facade
  participant L as Legacy system
  participant N as New service
  C->>R: GET /invoices/9
  R->>L: forward (still owner)
  R-)N: shadow copy
  L-->>R: invoice v1
  N--)R: invoice v1, compared and logged
  R-->>C: legacy response
  Note over R,N: after mismatch rate is near zero, flip route
  C->>R: POST /invoices
  R->>N: forward (new owner of writes)
  N-->>R: created
  N-)L: sync via CDC or events for legacy readers
  R-->>C: created

Numbers. Shadow traffic doubles load on the new path and on any shared dependency, so shadow a sample (say 10%) of a heavy endpoint. Comparisons need volume to be meaningful: to detect a 0.1% mismatch rate with reasonable confidence you need several thousand compared requests per variant (roughly 3 ÷ rate gives at least one expected mismatch at about 95% probability, so 3,000 at 0.1%). Each ramp step should run long enough to cover the daily traffic cycle.

Where it's used. Fowler's article is the canonical description, updated in 2024 with the idea of transitional architecture. GitHub's open source Scientist library runs the old and new code paths side by side and reports mismatches, which is the shadow-and-compare step in library form. AWS documents the pattern for modernizations.

Trade-offs and failure modes.

Sources: Fowler, Strangler Fig Application, AWS, Strangler fig, GitHub, Scientist.

Zero-downtime schema migration (expand and contract)

What it is. Change a database schema under live traffic by splitting one breaking change into a sequence of small deploys, each compatible with the one before it. You first expand the schema so old and new shapes coexist, migrate code and data across, then contract by removing the old shape. Fowler calls the general technique parallel change.

Why one deploy can't do it. A rolling deploy runs the old version N and the new version N+1 side by side for minutes, and a rollback brings N back. So every schema state must work with two adjacent application versions. Renaming email to email_address in one step breaks every N instance the moment the migration runs, and breaks N+1 if it runs late. The fix is never to change a shape in place: add the new one, move over, then delete the old one.

How it works. Take that rename as the example. Each row is a separate deploy, and you move to the next only when its gate passes:

Step Schema Writes go to Reads come from Gate before the next step
1. Expand Add email_address, nullable Old only (code unchanged) Old DDL applied on primary and all replicas
2. Dual write, then backfill Both columns Both, in one transaction Old Backfill done; a scan finds no rows where the columns differ
3. Switch reads Both columns New only (see below) New, falling back to old if new is null Read comparison shows no mismatches; no error spike
4. New only Both columns New New; the fallback to old is removed No code path, job or report touches the old column
5. Contract Archive, then drop email New New Done

Dual writes alone only cover rows written after step 2, so the backfill copies the history in small batches. Step 3's fallback read (COALESCE(email_address, email)) also covers a row the backfill hasn't reached yet.

The sharp edge in step 3. Step 3 changes reads and writes in the same deploy, and that breaks the "two adjacent versions" rule. During its rolling deploy, step-2 instances still read the old column while step-3 instances have stopped writing it, so some users see stale values. A rollback to step 2 has the same problem. The safe order splits the step: 3a switches reads to new while still writing both; 3b stops writing the old column once no step-2 instance remains. Stripe used this order: dual write, switch reads (verified by comparing old and new results with GitHub's Scientist), then switch writes, then remove old data.

flowchart TD
  E["1. Expand: add new column"] --> D["2. Dual write, read old"]
  D --> B["Backfill old rows in batches"]
  B --> V{"Columns match?"}
  V -- no --> B
  V -- yes --> R["3a. Read new, still write both"]
  R --> W["3b. Stop writing old"]
  W --> N["4. Remove fallback read"]
  N --> C["5. Archive, then drop old"]

sequenceDiagram
  participant N0 as App version N
  participant N1 as App version N+1
  participant DB as Database
  Note over N0,N1: rolling deploy, both versions serve traffic
  N0->>DB: write email (step 1 code)
  N1->>DB: write email and email_address in one transaction
  N0->>DB: read email: still correct
  N1->>DB: read email: still correct
  Note over N0,DB: every step keeps the previous version working

DDL that doesn't lock the table (PostgreSQL). Most ALTER TABLE forms take an ACCESS EXCLUSIVE lock. The real outage risk is the lock queue: an ALTER waiting behind one long-running query blocks every later query on that table. So:

SET lock_timeout = '2s';                              -- give up quickly, then retry
ALTER TABLE users ADD COLUMN email_address text;      -- metadata only, no rewrite

-- step 2: backfill in batches, between deploys
UPDATE users SET email_address = email
WHERE id >= 1000000 AND id < 1001000 AND email_address IS NULL;

-- before relying on the new column: constrain without a long lock
ALTER TABLE users ADD CONSTRAINT email_address_present
  CHECK (email_address IS NOT NULL) NOT VALID;        -- skips the full-table scan
ALTER TABLE users VALIDATE CONSTRAINT email_address_present;  -- weaker lock, writes continue
ALTER TABLE users ALTER COLUMN email_address SET NOT NULL;    -- scan skipped: valid CHECK proves it
CREATE UNIQUE INDEX CONCURRENTLY users_email_address ON users (email_address);

-- step 5, once nothing reads it
ALTER TABLE users DROP COLUMN email;

Adding a column with no default or a non-volatile default stores the default in metadata, so it doesn't rewrite the table. A volatile default such as clock_timestamp() does. On MySQL 8.0, ALGORITHM=INSTANT is the default for adding columns since 8.0.12, at any position since 8.0.29, and for dropping columns since 8.0.29. A table allows at most 64 instant row versions before it must be rebuilt. Changes MySQL can't do in place, such as type changes on big tables, use an online schema-change tool: pt-online-schema-change keeps a shadow table in sync with triggers, while GitHub's gh-ost tails the binlog instead, so it can pause entirely and lets you choose when to cut over.

Numbers.

Where it's used. Stripe moved hundreds of millions of subscription records from the customer table to their own table with this sequence, keeping all services at 100% availability and never changing more than a few hundred lines of code at once. GitHub built gh-ost for its MySQL fleet. The same expand and contract shape applies to APIs (add the new field, migrate consumers, remove the old one) and to message schemas.

Trade-offs and failure modes.

Sources: Fowler, Parallel Change, Stripe, Online migrations at scale, PostgreSQL ALTER TABLE, MySQL online DDL, gh-ost.

Control plane and data plane separation

What it is. Split the system into a control plane that makes changes (create, configure, scale, delete) and a data plane that serves the traffic using the resulting configuration. Design the data plane to keep working with its last known configuration when the control plane is down: static stability.

How it works. The control plane validates requests, stores desired state in a strongly consistent store, computes configuration and pushes (or lets data-plane nodes pull) versioned snapshots. Data-plane nodes cache the latest snapshot locally, serve every request from it, and never call the control plane on the request path. AWS's guidance adds: pre-provision the capacity you need to survive a failure, rather than relying on the control plane to launch it during the failure.

flowchart TB
  subgraph CP["Control plane (changes, low rate)"]
    API["Management API"] --> ST[("Desired state, consistent store")]
    ST --> CFG["Config compiler and distributor"]
  end
  subgraph DP["Data plane (traffic, high rate)"]
    N1["Node 1, cached config v42"]
    N2["Node 2, cached config v42"]
    N3["Node 3, cached config v41"]
  end
  CFG -->|push versioned snapshots| N1
  CFG --> N2
  CFG --> N3
  U["User traffic"] --> N1
  U --> N2

Numbers.

Where it's used. AWS describes EC2 this way: launching instances is control plane; routing packets for running instances and serving EBS reads and writes is data plane, which keeps working during control-plane impairment. AWS notes the control plane has more moving parts and is statistically more likely to be impaired, so the data plane targets higher availability. Kubernetes (API server and controllers vs kubelet and kube-proxy) and Envoy (xDS management server vs proxies) follow the same split.

Trade-offs and failure modes.

Sources: AWS Builders' Library, Static stability using Availability Zones, AWS, Fault isolation boundaries: control planes and data planes.

Cells and deployment stamps

What it is. Run many complete, independent copies of the service stack (cells, or deployment stamps in Azure terms), each serving a fixed subset of customers, with a thin router in front. A failure, bad deployment or overload stays inside one cell.

How it works. Each cell has its own compute, databases and queues, sized to a tested maximum. A cell router maps a partition key (tenant ID, account ID) to a cell via a mapping table and should be as simple as possible, since it is shared by everyone. Deployments roll out one cell at a time, starting with a small one. New customers go to new cells; a cell migration process moves tenants when a cell fills or a tenant grows. Shuffle sharding refines isolation: instead of fixed groups, each customer gets a random combination of k workers out of n, so two customers rarely share all their workers.

flowchart TB
  U["Requests with tenant ID"] --> RT["Cell router, thin, mapping table"]
  RT -->|tenants A-F| C1
  RT -->|tenants G-M| C2
  RT -->|tenants N-Z| C3
  subgraph C1["Cell 1"]
    A1["App"] --> D1[("DB")]
  end
  subgraph C2["Cell 2"]
    A2["App"] --> D2[("DB")]
  end
  subgraph C3["Cell 3"]
    A3["App"] --> D3[("DB")]
  end
  CPL["Control plane: placement, migration"] -.-> RT

Numbers.

Where it's used. AWS has used cell-based architecture in its services for more than a decade and published a Well-Architected guide to it in 2023. Slack migrated to a cellular architecture aligned to availability zones after a 2021 gray failure in one AZ, so traffic can be drained from a zone within 5 minutes; the project took about 1.5 years, with stateful services the hardest part. Azure documents the same idea as deployment stamps.

Trade-offs and failure modes.

Sources: AWS, Reducing the Scope of Impact with Cell-Based Architecture, AWS Builders' Library, Workload isolation using shuffle-sharding, Slack, Migration to a cellular architecture, Azure, Deployment Stamps.

Multi-region architecture

What it is. Running the system in more than one geographic region for disaster recovery, lower latency or data residency. The key decision is not where you serve traffic but where each record may be written.

How it works. Three common shapes:

Shape Writes Reads RPO RTO Main risk
Active-passive (warm standby) One primary region Primary; standby may serve stale reads Replication lag (seconds) Minutes: detect, promote, reroute Failover never tested; split brain if the old primary is not fenced
Active-active, home-region ownership Each user or tenant writes in its home region Local, with remote data stale Lag for the failed home's records Minutes to re-home Users who travel; re-homing during an outage
Active-active, multi-writer Any region Local Lag (async) or zero (sync quorum) Near zero Conflicts: last-writer-wins drops updates; sync adds WAN RTT to every write

Conflict handling for multi-writer data: last-writer-wins by timestamp (simple, silently loses concurrent updates), CRDTs for counters and sets, application merge functions, or avoid conflicts entirely by giving each record a single home region. Strong multi-region consistency needs a consensus quorum across regions (Spanner, DynamoDB's multi-region strong consistency mode), which costs at least one inter-region round trip per write.

flowchart LR
  DNS["Geo DNS or global load balancer"] --> R1
  DNS --> R2
  subgraph R1["Region us-east, home of US users"]
    A1["App"] --> D1[("Primary for home-us users")]
  end
  subgraph R2["Region eu-west, home of EU users"]
    A2["App"] --> D2[("Primary for home-eu users")]
  end
  D1 -.->|async replication, lag = RPO| D2
  D2 -.->|async replication| D1
  A2 -->|write for a home-us user, forwarded| A1

Numbers.

Where it's used. Facebook's memcache paper describes one master region holding the primary MySQL databases, with other regions serving reads from replicas and using "remote markers" to reduce stale reads after a user's own write. DynamoDB global tables offer both last-writer-wins eventual mode and a three-region strong mode with RPO = 0. Google Spanner uses Paxos across replicas and TrueTime for external consistency.

Trade-offs and failure modes.

Sources: DynamoDB global tables: how it works, Aurora Global Database failover, Nishtala et al., Scaling Memcache at Facebook, Corbett et al., Spanner.

Further reading

All chapters