gRPC From the Ground Up

gRPC has been in production at Google-scale since 2015 and has become the dominant RPC framework for polyglot microservice systems — but most introductions treat it as “REST with Protobuf,” which misses the point entirely. The protocol is built directly on HTTP/2, and that foundation is what enables multiplexed bidirectional streaming, first-class deadline propagation, and generated strongly-typed stubs across a dozen languages from a single schema. The tradeoff is real: binary framing, a non-trivial toolchain, load-balancing behavior that surprises engineers coming from stateless HTTP/1.1 services, and genuine friction in browsers. Understanding the transport layer is not optional background knowledge — it is the key to understanding why gRPC works the way it does, and why certain failure modes appear where they do.

If you are already working with the modern protobuf toolchain with Buf to manage your schemas, lint your APIs, and generate code, this post picks up where that one leaves off. It covers the wire protocol, the four call patterns, everything the runtime does for you around deadlines and cancellation, the interceptor model, the error taxonomy, and the load-balancing problem that trips up nearly every team shipping gRPC behind Kubernetes.

The HTTP/2 Foundation

gRPC does not define its own transport. Every gRPC call is an HTTP/2 request, and that is not an implementation detail — it is the architecture. To understand gRPC you need to understand the three HTTP/2 features it depends on.

Multiplexed streams over a single connection. HTTP/1.1 is a serial protocol. You can pipeline requests, but responses must arrive in order, and a slow response blocks everything behind it (head-of-line blocking). HTTP/2 replaces that model with independent numbered streams: each stream carries one request/response exchange, and frames from different streams are interleaved freely over the same TCP connection. A single gRPC channel between a client and server holds one TCP connection, and every concurrent RPC runs on its own stream. This eliminates the connection-pooling overhead that REST services manage implicitly and makes high-concurrency patterns cheap.

Binary framing and header compression. HTTP/2 is a binary protocol. Requests and responses are broken into frames — DATA frames carry payload, HEADERS frames carry metadata, and WINDOW_UPDATE frames implement per-stream flow control. Headers are compressed with HPACK, which matters for gRPC because every call carries :method, :path, content-type: application/grpc, grpc-timeout, and any custom metadata. On a high-RPC-rate service those header bytes add up; HPACK keeps them manageable.

Trailers. This is the feature HTTP/1.1 simply cannot do. gRPC communicates the final status of a call — the status code and any rich error details — in HTTP/2 trailer frames, which arrive after the response body is fully sent. That design is what makes it possible to stream an arbitrary number of response messages and then close the stream with a definitive success or error status. Any attempt to emulate gRPC over HTTP/1.1 requires proxying the trailers into the response body (the gRPC-Web encoding does exactly this, which is why it needs a proxy).

Each gRPC call maps to a single HTTP/2 stream. The request path encodes the service and method name: POST /package.ServiceName/MethodName. The request and response bodies are length-prefixed message frames: a 5-byte header (1 compression flag byte + 4 length bytes) followed by the serialized protobuf. Multiple messages in a streaming call are sent as sequential length-prefixed frames on the same HTTP/2 stream.

Client                          Server
  |                               |
  |  HTTP/2 HEADERS (stream 1)    |   :method POST
  |  :path /route.Router/Route    |   content-type: application/grpc
  |  grpc-timeout: 5000m          |   grpc-encoding: identity
  |------------------------------>|
  |                               |
  |  HTTP/2 DATA (stream 1)       |   [5-byte length prefix][protobuf payload]
  |------------------------------>|
  |                               |
  |  HTTP/2 HEADERS (stream 1)    |   :status 200
  |<------------------------------|   content-type: application/grpc
  |                               |
  |  HTTP/2 DATA (stream 1)       |   [5-byte length prefix][protobuf response]
  |<------------------------------|
  |                               |
  |  HTTP/2 TRAILERS (stream 1)   |   grpc-status: 0
  |<------------------------------|   grpc-message: ""

The grpc-status trailer is the canonical success/failure signal. An HTTP 200 with a non-zero grpc-status means the RPC failed at the application layer. This is frequently misunderstood when debugging through infrastructure that only inspects HTTP status codes.

The Four Call Types

gRPC defines four RPC patterns. The .proto syntax makes them explicit, and the generated stubs enforce them at compile time.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

syntax = "proto3";

package orders.v1;

option go_package = "github.com/example/orders/gen/go/orders/v1;ordersv1";

message CreateOrderRequest {
  string customer_id = 1;
  repeated string product_ids = 2;
}

message CreateOrderResponse {
  string order_id = 1;
}

message GetOrderRequest {
  string order_id = 1;
}

message Order {
  string order_id = 1;
  string status = 2;
  int64  created_at = 3;
}

message UploadItemsRequest {
  string product_id = 1;
  int32  quantity = 2;
}

message UploadItemsResponse {
  int32 items_received = 1;
}

message WatchOrdersRequest {
  string customer_id = 1;
}

service OrderService {
  // Unary: one request, one response
  rpc CreateOrder(CreateOrderRequest) returns (CreateOrderResponse);

  // Server streaming: one request, stream of responses
  rpc GetOrderHistory(GetOrderRequest) returns (stream Order);

  // Client streaming: stream of requests, one response
  rpc UploadInventory(stream UploadItemsRequest) returns (UploadItemsResponse);

  // Bidirectional streaming: both sides stream independently
  rpc WatchOrders(stream WatchOrdersRequest) returns (stream Order);
}

Unary is the right default. Most RPCs are request/response: create this thing, fetch that thing. Server streaming makes sense when the result set is unbounded or large enough that buffering it in a single response is wasteful — think a query that returns tens of thousands of rows, or a subscription to a live event feed. Client streaming suits aggregation workloads where you are uploading a batch that the server processes as a unit. Bidirectional streaming is the most powerful and the most operationally complex; it is easy to misuse, difficult to reason about at the protocol level, and requires careful handling of half-close semantics (the client closes its send side, the server drains the remaining work, and then closes its send side). Reach for it only when you genuinely have two independent message flows that need to proceed concurrently over the same logical session.

Generating Code

With Buf, generating stubs from the above schema is a buf.gen.yaml and a single command. See the modern protobuf toolchain with Buf for the full workflow; the minimal incantation for Go looks like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# buf.gen.yaml
version: v2
plugins:
  - remote: buf.build/protocolbuffers/go
    out: gen/go
    opt:
      - paths=source_relative
  - remote: buf.build/grpc/go
    out: gen/go
    opt:
      - paths=source_relative

1

buf generate

The output is two files per package: *.pb.go (message types) and *_grpc.pb.go (service client and server interfaces). The server interface requires implementing one method per RPC; the client struct is ready to use immediately.

Deadlines and Cancellation

Deadline propagation is one of the most operationally important features in gRPC and one that REST/JSON services almost never get right. The model is simple: the client sets a deadline when it creates the call context, that deadline is encoded in the grpc-timeout header as the remaining duration, and any downstream calls the server makes should be bounded by the same deadline with the elapsed time subtracted. When the deadline expires, the client receives DEADLINE_EXCEEDED, and the server’s context is cancelled — if the handler is checking ctx.Err() and passing the context to its own outgoing calls, the cancellation propagates through the entire call chain automatically.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

// Client: set a 2-second deadline on the call
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()

resp, err := client.CreateOrder(ctx, &ordersv1.CreateOrderRequest{
    CustomerId: "cust-123",
    ProductIds: []string{"prod-A", "prod-B"},
})
if err != nil {
    st, _ := status.FromError(err)
    switch st.Code() {
    case codes.DeadlineExceeded:
        log.Printf("call timed out after 2s")
    case codes.Canceled:
        log.Printf("call was cancelled")
    default:
        log.Printf("RPC error: %v", err)
    }
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

// Server: propagate the context (and its deadline) to downstream calls
func (s *orderServer) CreateOrder(ctx context.Context, req *ordersv1.CreateOrderRequest) (*ordersv1.CreateOrderResponse, error) {
    // ctx already carries the client's deadline; pass it downstream
    inv, err := s.inventoryClient.Reserve(ctx, &inventoryv1.ReserveRequest{
        ProductIds: req.ProductIds,
    })
    if err != nil {
        return nil, err // propagates DEADLINE_EXCEEDED or UNAVAILABLE
    }
    _ = inv
    return &ordersv1.CreateOrderResponse{OrderId: "ord-456"}, nil
}

The contrast with REST is stark. In a REST service chain, each hop typically sets its own timeout independently (or worse, has no timeout at all). A 500ms client timeout at the edge means nothing if the middle-tier service has a 30-second default. With gRPC’s context model, a 500ms grpc-timeout at the edge becomes at most 500ms minus network latency for every downstream hop. This is what actually prevents cascading latency failures.

gRPC also supports explicit client-side cancellation via cancel(). A client that abandons a streaming result set can call cancel and the server handler’s context will become done, which — if the handler respects the context — stops work immediately rather than running to completion for a result nobody will consume.

Interceptors

Interceptors are gRPC’s middleware model. There are four distinct types: unary server, unary client, stream server, stream client. Each type has a distinct interface because the streaming variant needs to wrap a stream object rather than a single handler invocation.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

// Unary server interceptor: logging + request ID injection
func loggingInterceptor(
    ctx context.Context,
    req interface{},
    info *grpc.UnaryServerInfo,
    handler grpc.UnaryHandler,
) (interface{}, error) {
    start := time.Now()
    resp, err := handler(ctx, req)
    log.Printf("method=%s duration=%s err=%v", info.FullMethod, time.Since(start), err)
    return resp, err
}

// Wire it up with chaining (order matters: first is outermost)
s := grpc.NewServer(
    grpc.ChainUnaryInterceptor(
        authInterceptor,
        loggingInterceptor,
        metricsInterceptor,
    ),
    grpc.ChainStreamInterceptor(
        authStreamInterceptor,
        loggingStreamInterceptor,
    ),
)

The grpc-ecosystem/go-grpc-middleware library provides production-ready interceptors for OpenTelemetry tracing, Prometheus metrics, structured logging (with per-field extractors for request IDs and tenant IDs), and retry with backoff. Interceptors are compositional: add auth once at the server level and every RPC is protected, no per-handler boilerplate required.

The critical difference between unary and stream interceptors is that a stream interceptor receives a grpc.ServerStream interface; to inspect or modify individual messages you must wrap that interface and override RecvMsg and SendMsg. This is more verbose than unary interceptors and easy to get wrong. The middleware library handles this correctly; implementing a custom message-logging stream interceptor from scratch is a common source of subtle bugs.

The Error Model

gRPC defines a fixed set of status codes, and the mapping to them is part of your service contract.

The status code travels in the grpc-status trailer. The grpc-message trailer holds a human-readable string, but that string is for developers, not for programmatic error handling. For structured machine-readable error details, use google.rpc.Status from the google.golang.org/genproto/googleapis/rpc/status package, which lets you attach typed protobuf messages to the error: BadRequest with a list of field violations, RetryInfo with a suggested backoff duration, ErrorInfo with a domain and reason code for structured logging, and QuotaFailure with specific quota names.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

import (
    "google.golang.org/grpc/status"
    "google.golang.org/grpc/codes"
    epb "google.golang.org/genproto/googleapis/rpc/errdetails"
)

func (s *orderServer) CreateOrder(ctx context.Context, req *ordersv1.CreateOrderRequest) (*ordersv1.CreateOrderResponse, error) {
    if req.CustomerId == "" {
        st, _ := status.New(codes.InvalidArgument, "validation failed").
            WithDetails(&epb.BadRequest{
                FieldViolations: []*epb.BadRequest_FieldViolation{
                    {Field: "customer_id", Description: "must not be empty"},
                },
            })
        return nil, st.Err()
    }
    // ...
    return &ordersv1.CreateOrderResponse{OrderId: "ord-789"}, nil
}

Comparing gRPC’s error model to REST HTTP status codes is instructive. DEADLINE_EXCEEDED maps to nothing in HTTP — the closest is 504 Gateway Timeout, but that only means the gateway timed out, not the application. FAILED_PRECONDITION vs UNAVAILABLE is a distinction REST has no built-in way to make: the former means “the server is fine, but this operation cannot proceed given current state” (do not retry), while the latter means “the server is temporarily overloaded” (retry with backoff). gRPC clients that implement retry logic can use this distinction to avoid hammering a service that is returning FAILED_PRECONDITION on bad input, while still retrying on UNAVAILABLE.

Load Balancing: The Hard Part

This is where gRPC surprises teams used to HTTP/1.1 services. Because gRPC holds one long-lived HTTP/2 connection per channel, an L4 load balancer — including a default Kubernetes Service — sees exactly one TCP connection per client. All multiplexed RPCs on that connection go to a single backend pod. You can scale to fifty replicas and find that one of them is handling 100% of traffic.

The L4 problem:

  gRPC client
      |
      | (one TCP connection, all streams)
      v
  [L4 LB / kube-proxy]
      |
      | (pinned to one backend — L4 sees TCP, not HTTP/2 streams)
      v
  backend-pod-3     <- 100% of load
  backend-pod-1     <- idle
  backend-pod-2     <- idle
  backend-pod-4     <- idle

The L7 / client-side solution:

  gRPC client
   | resolver (DNS / xDS) returns N endpoints
   |
   +--stream 1--> backend-pod-1
   +--stream 2--> backend-pod-2
   +--stream 3--> backend-pod-3
   +--stream 4--> backend-pod-4
      (round-robin or weighted, per-RPC)

There are three viable solutions:

Client-side load balancing with a name resolver. The gRPC client has a pluggable resolver/balancer interface. With a DNS resolver that returns all A records for a headless Kubernetes Service (one A record per pod IP), and a round-robin load balancing policy, the client opens one connection to each pod and distributes RPCs across them. The downside is that the client needs to know about all backend addresses, and re-resolution on pod churn requires a brief reconnect window.

L7 proxy (Envoy, or a service mesh). An L7 proxy understands HTTP/2. It terminates the client’s connection, and opens independent connections to each backend, routing each RPC independently. In Kubernetes, adding Istio or Linkerd gives you this automatically — the Envoy sidecar intercepts outbound gRPC and does request-level load balancing. See the discussion in Service Meshes with Istio and Linkerd for the control-plane side of this story.

Proxyless xDS (gRPC-native service mesh). gRPC’s most architecturally interesting load-balancing story: the gRPC client embeds an xDS client (the same protocol Envoy uses to receive configuration from Istio’s control plane). The application connects to a control plane — Google’s Traffic Director, Istio’s Istiod, or any xDS-compatible server — using a boot configuration file and an xds:/// resolver scheme. The control plane pushes endpoint lists, load balancing policy, and routing config directly to the application. No sidecar proxy, no extra network hop, no Envoy process to size and manage.

1
2
3
4
5
6

// Proxyless xDS channel — the "xds:///" scheme triggers the embedded xDS client
// GRPC_XDS_BOOTSTRAP env var points to the bootstrap JSON file
conn, err := grpc.NewClient(
    "xds:///orders.prod.svc.cluster.local",
    grpc.WithTransportCredentials(credentials.NewTLS(&tls.Config{})),
)

The boot file tells the gRPC process where to find the control plane and its own node identity so the control plane knows which service it is configuring. Proxyless xDS supports round-robin and ring-hash load balancing policies in current implementations. More complex policies (least-request, etc.) are only available through an L7 proxy today, and the setup is meaningfully more complex than a sidecar-based mesh. But for latency-critical paths where you cannot afford the sidecar hop, it is the right architecture.

Reflection and Tooling

gRPC server reflection is a standardized protocol (itself a gRPC service) that lets clients query a running server for its service definitions, method signatures, and message schemas without having the .proto files locally. It is essential for debugging tools.

grpcurl is the standard command-line tool:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# List services on a server with reflection enabled
grpcurl -plaintext localhost:50051 list

# Describe a specific service
grpcurl -plaintext localhost:50051 describe orders.v1.OrderService

# Invoke a unary RPC with a JSON payload
grpcurl -plaintext -d '{"customer_id": "cust-123", "product_ids": ["prod-A"]}' \
  localhost:50051 orders.v1.OrderService/CreateOrder

# Watch a server-streaming RPC
grpcurl -plaintext -d '{"customer_id": "cust-123"}' \
  localhost:50051 orders.v1.OrderService/WatchOrders

evans is a REPL-style interactive gRPC client that supports reflection, is useful for exploratory testing, and has a more ergonomic interface for services with many methods.

buf curl is Buf’s alternative that integrates with the Buf Schema Registry — so you can invoke RPCs against a server that does not have reflection enabled at all, as long as the schema is published to BSR. This is safer for production servers where you may not want to expose the full schema via reflection.

Enable reflection in Go during development:

1
2
3
4
5

import "google.golang.org/grpc/reflection"

s := grpc.NewServer(/* interceptors */)
ordersv1.RegisterOrderServiceServer(s, &orderServer{})
reflection.Register(s) // disable in production or gate behind a flag

gRPC-Web and Connect: Calling gRPC from Browsers

Browsers cannot speak raw gRPC. The fetch API and XMLHttpRequest do not expose HTTP/2 framing, and — critically — browsers have no mechanism to read HTTP/2 trailer frames. Without trailers, the grpc-status signal cannot reach the application layer.

gRPC-Web solves this by encoding trailers into the response body (a message with a distinguished flag byte in the 5-byte frame header). The browser’s gRPC-Web client library decodes them. A proxy (Envoy, or the grpcwebproxy tool) sitting in front of the gRPC server translates between gRPC-Web requests from browsers and standard gRPC on the backend. The mandatory proxy is an operational burden; gRPC-Web also has gaps in streaming support depending on which proxy and client library you use.

Connect (connectrpc.com) takes a different approach. The Connect protocol is a first-class HTTP/1.1 and HTTP/2 protocol for Protobuf RPCs. A Connect server speaks three protocols simultaneously: standard gRPC, gRPC-Web, and the Connect protocol. The Connect protocol encodes unary calls as plain HTTP POST requests with JSON or binary Protobuf bodies and surface-level HTTP status codes, which means a Connect unary call is inspectable with curl, readable in browser devtools, and requires no translation proxy. Streaming in the Connect protocol works over HTTP/2 (using standard HTTP/2 streams, not trailer-in-body encoding).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

// connect-es: TypeScript client calling a Connect server from the browser
import { createClient } from "@connectrpc/connect";
import { createConnectTransport } from "@connectrpc/connect-web";
import { OrderService } from "./gen/orders/v1/orders_connect";

const transport = createConnectTransport({ baseUrl: "https://api.example.com" });
const client = createClient(OrderService, transport);

const res = await client.createOrder({
  customerId: "cust-123",
  productIds: ["prod-A", "prod-B"],
});
console.log(res.orderId);

connect-go servers are wire-compatible with gRPC clients and vice versa; you can deploy a connect-go server and have Go gRPC clients, TypeScript browser clients, and Python gRPC clients all communicating with the same backend. For new services where browser access is a requirement, Connect is now the pragmatic default over gRPC-Web.

For context on where gRPC and Connect fit in the broader API design space, compare with REST API design principles and OpenAPI and Swagger — the schema-first contract approach of gRPC overlaps with and competes with OpenAPI-driven REST in interesting ways.

Downsides and When to Use Something Else

The honest accounting:

Binary wire format. A gRPC exchange is not human-readable. You cannot curl a gRPC endpoint and read the response in your terminal. Debugging requires grpcurl, a service mesh observability tool that can decode Protobuf, or enabling a transcoding proxy. Teams that want to inspect traffic at a network level need additional tooling investment.

Browser friction. As described above, raw gRPC from a browser is not possible without gRPC-Web (proxy required) or Connect (no proxy, but a different protocol). If your primary consumers are browsers, start with Connect from the beginning.

Toolchain and codegen complexity. Every client and server requires a code generation step. That step must be integrated into every build system, every language runtime, and every CI pipeline. Schema changes require regenerating stubs across all consumers. The Buf toolchain has made this significantly less painful than it was with raw protoc, but the inherent complexity is still higher than writing a JSON REST client with a hand-rolled struct.

Load balancing surprises. As detailed above: any team shipping gRPC into Kubernetes without understanding the L4/L7 problem will eventually encounter lopsided pod utilization. It is not a hard problem once you understand it, but it will bite you if you assume gRPC works like HTTP/1.1 behind a standard load balancer.

Debugging and observability. Binary frames in tcpdump or Wireshark are not readable without Protobuf schema awareness. Logs that print the raw error string from a gRPC status are less useful than structured JSON error responses. You need to invest in observability — structured logging from interceptors, OpenTelemetry traces, and Prometheus metrics from the grpc-ecosystem middleware — to have the same debuggability you get for free with JSON REST.

When REST/JSON is the better choice. If your consumers are external developers you do not control, REST with OpenAPI documentation is still the right default — the tooling ecosystem, caching semantics, and human-readability of JSON matter enormously at a public API boundary. If your service has simple CRUD semantics with no streaming requirements and is consumed by browsers, Connect-over-JSON is now a credible middle path. If your team is small and moving fast, the codegen step and schema management overhead may not be worth the benefits until you have multiple language consumers or genuine high-throughput streaming requirements.

On the emerging transport front: gRPC over HTTP/3 (QUIC) is a subject of active work in the grpc-core and grpc-go projects. QUIC eliminates TCP’s head-of-line blocking at the transport layer (which HTTP/2 multiplexing still suffers from at the TCP level under packet loss) and improves connection establishment time. Production support is not yet stable across all gRPC implementations as of mid-2026, but it is the likely next step for the transport layer, and worth watching for latency-sensitive services.

Verdict

gRPC is the right framework for internal service-to-service communication at scale: polyglot teams, high-throughput bidirectional streaming, deadline propagation across a call chain, and strong schema contracts enforced at compile time. The HTTP/2 foundation is what makes all of this possible and is worth understanding directly rather than treating as an abstraction. The load balancing story requires deliberate design — either client-side resolution, an L7 proxy, or proxyless xDS — and will not work correctly with naive Kubernetes Service routing.

For browser-facing APIs, Connect is the pragmatic answer in 2026: full gRPC wire compatibility on the server, no proxy requirement, readable in devtools, and TypeScript/Kotlin/Swift code generation from the same .proto schema. For public-facing APIs with external developers, REST with OpenAPI documentation remains the more accessible choice. For everything in the internal mesh, gRPC with a Buf-driven schema workflow is a well-understood, production-proven stack.

Sources

• gRPC on HTTP/2 — Engineering a Robust, High-performance Protocol
• gRPC Load Balancing — official blog
• Deadlines — gRPC documentation
• Status Codes — gRPC documentation
• Error Handling — gRPC documentation
• Reflection — gRPC documentation
• How Proxyless gRPC Works in a Service Mesh — gRPConf India 2025
• Keynote: What’s New in gRPC? — gRPConf India 2025
• Cloud Service Mesh with proxyless gRPC services overview — Google Cloud
• How to Use Istio with gRPC Proxyless Service Mesh
• Connect RPC — Introduction
• Choosing a protocol — Connect RPC
• connect-go — GitHub
• gRPC in Go: Streaming RPCs, Interceptors, and Metadata — VictoriaMetrics blog
• go-grpc-middleware — GitHub
• GRPC Core: gRPC over HTTP2 — protocol reference
• Don’t Load Balance gRPC or HTTP/2 Using Kubernetes Service
• Introducing buf curl — Buf blog