LUNAROPS · OPERATIONAL UPLINK 100% UPTIME 1,247d POSTS 893 JEFF.MOON@LUNAROPS.DEV UTC --:--:--

Microservices Patterns That Work

14 min read Software Architecture
microservicesdistributed-systemsresiliencesaga-patternapi-gatewayarchitecture
Contents

Microservices are not an architecture you adopt because they are modern; they are a trade you make because an organization has grown past what a single deployable unit can absorb. The honest framing, which most enthusiastic adoptions skip, is that you are exchanging one category of difficulty for another. A monolith’s problems are real but legible: a tangled call graph, a shared database everyone is afraid to migrate, a deploy that takes the whole app down. Microservices dissolve those and hand you a different bag — every in-process function call that used to either succeed or throw is now a network request that can also hang, time out, partially succeed, or succeed-but-the-response-got-lost. You have traded a complexity you can see in a stack trace for a complexity that lives in the gaps between services, and the patterns in this piece exist to make that second category survivable.

The reason to make the trade is almost always organizational before it is technical. Microservices let independent teams own, deploy, and scale their pieces on their own cadence without coordinating a monolithic release — that is the actual payoff, and it is a people-and-process win that happens to require a technical architecture. If you do not have multiple teams stepping on each other, or independent scaling needs, or a domain with genuinely separable boundaries, you are likely buying the costs without the benefit. So before any pattern, the first and most important judgment is whether to split at all.

How services talk: synchronous versus asynchronous

The most consequential decision in a microservice system is how services communicate, because it determines how failure propagates. There are two fundamental shapes, and conflating them is the root of most distributed-system pain.

Synchronous communication — REST over HTTP, or gRPC — is a request the caller blocks on, waiting for a response. It is simple to reason about and right for queries that need an immediate answer: the checkout page asks the inventory service “is this in stock?” and cannot proceed without the answer.

1
2
3

# Synchronous: caller blocks, and inherits every failure mode of the callee.
resp = http.get("http://inventory/v1/stock/SKU-1", timeout=0.5)
in_stock = resp.json()["available"]

The hidden cost is temporal coupling: the caller is only as available as the callee. Chain five synchronous calls and your effective availability is the product of all five, while your latency is their sum. One slow dependency stalls the whole chain. This is why synchronous calls want aggressive timeouts and a circuit breaker around them.

Asynchronous communication — publishing an event to a broker that other services consume on their own schedule — decouples services in time. The order service announces “order.created” and moves on; the email service, the analytics service, and the warehouse service each react when they get to it, and if one is down the message waits.

1
2

# Asynchronous: fire-and-forget; consumers react when able, outages absorbed by the broker.
broker.publish("orders", "order.created", json.dumps(order))

This is the foundation of resilience, because a consumer being down no longer breaks the producer. The cost is that you give up the immediate, strongly-consistent answer — the system becomes eventually consistent, and you must design the UX and the data model to tolerate the lag. The deep treatment of brokers, delivery guarantees, and ordering lives in event-driven architecture; the rule of thumb is: synchronous for queries that need an answer now, asynchronous for everything that can tolerate “soon,” and bias toward asynchronous wherever you can, because it is what stops failures from cascading.

Aspect Synchronous (HTTP/gRPC) Asynchronous (events)
Coupling Temporal — caller needs callee up now Decoupled in time — broker buffers
Consistency Immediate answer possible Eventual; design for lag
Failure mode Cascades up the call chain Absorbed by the broker; consumer retries
Best for Queries needing an answer now Notifications, workflows, fan-out
Hard part Timeouts, retries, circuit breaking Idempotency, ordering, debugging flow

The edge: API gateway and the cross-cutting concerns

Exposing dozens of services directly to clients is untenable — every client would need to know every service’s address, and every service would re-implement authentication, rate limiting, and TLS. The API gateway is the single front door: clients talk to it, and it routes to the services behind, handling the cross-cutting concerns once, at the edge.

                     +------------------- API GATEWAY -------------------+
   client  ----->    |  TLS · authn · rate limit · routing · aggregation |
                     +---+----------------+-------------------+----------+
                         |                |                   |
                    [ user svc ]    [ order svc ]availability [ catalog svc ]

The gateway is the right home for authentication (verify the token once, pass a trusted identity inward), rate limiting, TLS termination, request routing, and sometimes response aggregation — fanning one client request out to several services and composing the results so the client makes one call instead of six. The discipline is to keep it thin: a gateway that accumulates business logic becomes a new monolith that every team must coordinate to change, recreating the exact bottleneck microservices were meant to remove. Routing and policy belong there; domain logic does not.

A close relative is the backend-for-frontend (BFF): a dedicated gateway per client type, because a mobile app and a web app want different shapes and granularities of data. Rather than bloat one gateway with every client’s needs, give each client its own thin aggregation layer.

Keeping failure contained: timeouts, retries, circuit breakers, bulkheads

In a monolith, a slow function is a slow function. In a distributed system, a slow service is a contagion: callers pile up waiting on it, exhausting their own thread and connection pools, until they become slow too, and the slowness propagates up the call graph until the whole system is wedged. This cascading failure is the characteristic way distributed systems die, and four patterns work together to contain it.

Timeouts are first and non-negotiable: every network call must have one, because the default in most clients is “wait forever,” and “forever” is how one stuck dependency takes down everything upstream. Retries handle transient blips — but only with exponential backoff and jitter, and only for idempotent operations, or you turn a brief hiccup into a self-inflicted retry storm that finishes off the recovering service. Circuit breakers stop hammering a service that is clearly down: after a threshold of failures the breaker “opens” and fails fast for a cooldown, sparing both the caller (no piling up on doomed calls) and the callee (room to recover), then “half-opens” to test the water before resuming.

1
2
3
4
5
6
7

from circuitbreaker import circuit

@circuit(failure_threshold=5, recovery_timeout=30)
def get_recommendations(user_id):
    # After 5 consecutive failures the breaker opens: calls fail fast for 30s
    # instead of stacking up on a dead service.
    return http.get(f"http://recs/v1/users/{user_id}", timeout=0.3).json()

   CLOSED  --(failures exceed threshold)-->  OPEN
     ^                                         |
     |                                  (cooldown elapses)
     |                                         v
     +------(trial succeeds)------------  HALF-OPEN  --(trial fails)--> OPEN

Bulkheads isolate resources so one failing dependency cannot drain the pool that healthy ones share — separate connection pools or thread pools per downstream, named after a ship’s compartments that keep one breach from sinking the whole hull. Together these four turn “one service died” from a system-wide outage into a localized, degraded-but-alive condition — ideally with a graceful degradation fallback so the breaker-open path returns cached or default data rather than an error. None of this is free: each adds latency budget, configuration, and failure modes of its own, and tuning timeout-and-retry values badly causes outages as surely as omitting them.

Finding each other: service discovery and the mesh

Services in a dynamic environment do not have fixed addresses — instances come and go as they scale and reschedule, so hardcoding IPs is impossible. Service discovery is how a service finds a current, healthy address for the thing it wants to call. In Kubernetes this is built in: a Service object gives a stable DNS name and virtual IP that load-balances across the live pods behind it, so callers just address user-service and the platform handles the rest.

1
2
3
4
5
6
7
8
9

apiVersion: v1
kind: Service
metadata:
  name: user-service
spec:
  selector:
    app: user-service     # routes to whatever pods currently carry this label
  ports:
    - port: 80

As the number of services grows, the resilience concerns above — timeouts, retries, circuit breaking, mTLS, tracing — start getting re-implemented in every service in every language, inconsistently. The service mesh (Istio, Linkerd) is the response: push all of that into a sidecar proxy deployed beside each service, so the platform handles retries, mutual TLS, traffic shifting, and telemetry uniformly, outside the application code. The deep trade-offs of running one live in service meshes: Istio vs Linkerd; the short version is that a mesh buys consistency and observability at the cost of real operational complexity and a proxy hop on every call, and a small system is better served by a resilience library than by a mesh.

The sidecar pattern generalizes beyond meshes: any helper that runs alongside the main container — a log shipper, a config reloader, a secrets agent — shares the pod’s lifecycle and network without being baked into the application image.

1
2
3
4
5
6

spec:
  containers:
    - name: app
      image: myapp
    - name: envoy          # sidecar: handles mTLS, retries, telemetry for `app`
      image: envoyproxy/envoy

The hardest part: data and distributed transactions

The pattern that breaks the most teams is data. The rule that makes microservices actually independent is database-per-service: each service owns its data and no other service touches its tables directly — they go through its API. The moment two services share a database, they are coupled at the schema, can no longer deploy or migrate independently, and you have a distributed monolith with all the costs of distribution and none of the autonomy. But honoring that rule means a business operation spanning several services — place an order, charge a card, reserve stock — can no longer be a single ACID transaction, because there is no shared database to wrap it in.

The saga pattern is the answer: model the cross-service operation as a sequence of local transactions, each publishing an event that triggers the next, and each paired with a compensating action that semantically undoes it if a later step fails. There is no rollback across services — there is forward recovery through compensation.

  Order Svc        Payment Svc        Inventory Svc
     |                  |                   |
  create order ---> charge card ----> reserve stock
     |                  |                   X  (out of stock!)
     |                  |                   |
     |<-- refund <------|<-- compensate ----+
   cancel order      (undo charge)     (nothing to undo)

Sagas come in two flavors with a real trade-off. Choreography has each service react to events with no central coordinator — simple and decoupled, but the overall workflow exists nowhere explicitly and becomes hard to follow as steps multiply. Orchestration introduces a coordinator that explicitly drives the steps — easier to understand, monitor, and modify, at the cost of a central component. The critical and frequently-botched detail is that compensations must be idempotent and able to handle “undo something that may not have fully happened,” because in a distributed system you often cannot be sure how far a failed step got. Designing those compensations is genuinely hard, and it is the tax you pay for giving up distributed transactions.

Two adjacent patterns recur in this space. The outbox pattern solves the “I updated my database and published an event, but the process died between them” problem by writing the event into the same database transaction as the state change and relaying it afterward — covered in depth in the outbox pattern. And event sourcing with CQRS — storing the sequence of events as the source of truth and projecting read-optimized views from them — is a powerful but heavyweight option for domains that need a complete audit trail and temporal queries, explored in event sourcing and CQRS. Both are tools for the same underlying reality: in a distributed system, consistency is something you design for, not something you get for free.

You cannot operate what you cannot see

A monolith’s failure leaves one stack trace in one log. A distributed request might touch eight services, and when it fails, the error surfaces in one of them while the cause lives three hops away. Without deliberate instrumentation, debugging becomes archaeology across eight separate log streams with no thread connecting them. Observability is therefore not optional in microservices — it is the price of admission, and the single most common reason teams that adopt microservices regret it is that they did the split without building the visibility to operate the result.

The load-bearing pattern is distributed tracing: a trace ID generated at the edge and propagated through every hop, so you can reconstruct a single request’s entire journey across services and see exactly which span was slow or threw — the subject of distributed tracing with Tempo and Grafana. Alongside it, structured logs that all carry the trace ID stitch the eight streams into one story, and per-service metrics (error rate, latency percentiles, saturation) tell you whether a problem is one instance or systemic. This is why “do you have the DevOps capability” is a real precondition for microservices, not a nice-to-have: the architecture relocates complexity from inside the code to the spaces between services, and observability is the only way to see into those spaces.

When not to split: the modular monolith

The most valuable microservices pattern is frequently the decision to not use them yet. Microservices impose a fixed cost — network failure handling, distributed data, deployment pipelines per service, service discovery, tracing, the whole apparatus above — that is only worth paying when the benefits (independent team autonomy, independent scaling, fault isolation) exceed it. For a small team, a clear-but-simple domain, or boundaries you do not yet understand, that math is firmly negative, and splitting early is how teams end up with a distributed monolith: services so chatty and co-dependent that they must be deployed together, combining the operational pain of distribution with the coupling of a monolith.

The strongly recommended default is to start with a modular monolith: a single deployable unit, but internally organized into well-bounded modules with explicit interfaces between them, as if they might one day be separate services. You get the operational simplicity of one deploy, one database transaction, one log, one debugger session — while keeping the architectural seams that let you extract a service later, when a real pressure (this module needs to scale separately, this team needs to ship independently) tells you which service to extract and where the boundary actually is. Boundaries discovered under real load are far better than boundaries guessed up front, and the cost of being wrong is a function move inside one process rather than a painful distributed-data migration. Extract services when you feel a specific pain that a service solves — not on principle, and not because the architecture diagrams look more impressive with boxes and arrows.

Verdict

Microservices are a trade, not an upgrade. They dissolve the legible problems of a monolith and hand you the harder, less visible problems of a distributed system — network partial failure, cascading slowness, data without distributed transactions, and operational sprawl — and the patterns here exist solely to make that trade survivable. Communicate asynchronously wherever you can so failures are absorbed rather than propagated; put cross-cutting concerns at a thin gateway; contain failure with timeouts, retries, circuit breakers, and bulkheads so one dead service degrades instead of detonates; give each service its own data and stitch cross-service operations together with sagas and the outbox rather than wishing for a distributed transaction; and instrument everything with tracing, because you cannot operate what you cannot see.

The meta-pattern beneath all of them: microservices relocate complexity from inside a process to the network between processes, and every pattern is a tool for managing complexity in that new location. That relocation is worth it when organizational scale demands independent teams and independent scaling — and a net loss otherwise. So the most important pattern is the discipline to start with a modular monolith and extract services only when a concrete pain points to a concrete boundary. Microservices are a tool for a specific problem, not a destination; reach for the pattern that solves the pain you actually have, and resist the ones that solve pains you have only read about.

Sources

Comments