KV Cache Engineering: PagedAttention, Continuous Batching, Attention Variants, and the Bandwidth Wall

The KV cache is responsible for more wasted GPU memory in LLM serving than any other single factor. Before PagedAttention, systems pre-allocated memory for the maximum possible sequence length for every request — even short conversations claimed the same memory footprint as the longest possible output. That pattern wasted 60–80% of GPU memory that could have been used to serve more concurrent requests.

Understanding the KV cache is not optional if you’re running or building LLM inference infrastructure. It determines how many concurrent requests you can serve, how long your contexts can be, what batch sizes are achievable, and whether prefix caching is giving you the 10× cost reduction it promises or producing a 2% hit rate that helps nobody.

This post covers the mechanics from the ground up: what the cache stores and why, how PagedAttention restructured memory allocation to eliminate fragmentation, how prefix caching reuses computation across requests with shared prefixes, and how FP8/INT8 quantization of the cache halves the memory footprint again with minimal accuracy cost.

What the KV Cache Stores and Why

In a Transformer, each attention layer computes query (Q), key (K), and value (V) projections from the input token embeddings. During autoregressive generation — where the model produces one token at a time, each new token attending to all previous tokens — the K and V projections for every previous token are needed again at each generation step.

Without caching, the model would recompute K and V for all previous tokens on every generation step. For a 1000-token context, generating token 1001 would recompute K and V for all 1000 tokens before computing the new one — O(n²) total compute for a sequence of length n.

The KV cache stores the K and V tensors for every token that has already been processed, so each generation step only computes K and V for the new token and reads the cached values for all previous tokens:

Without KV cache (step 1001 of generation):
  Process tokens 1..1001 → attend over all → generate token 1002
  Cost: O(1001) attention operations

With KV cache (step 1001 of generation):
  Read cached K,V for tokens 1..1000
  Compute K,V for token 1001 only
  Attend new Q over all cached K,V
  Cost: O(1) new computation + O(1000) read

Total cost with cache: O(n) over full generation
Total cost without:   O(n²) over full generation

The trade-off is memory for compute. The cache grows linearly with sequence length and must be held in GPU memory for the duration of the request. For a large model with many layers and a long context, this is substantial.

Cache size calculation

For a single request:

KV cache per token = 2 × n_layers × n_kv_heads × head_dim × dtype_bytes

Example: Llama 3.1 8B
  n_layers = 32
  n_kv_heads = 8  (GQA — grouped query attention)
  head_dim = 128
  dtype = BF16 (2 bytes)

KV cache per token = 2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KB

For a 4096-token context:
  4096 × 128 KB = 512 MB per request

For a larger model like Llama 3.1 70B (with 64 layers, 8 GQA heads):

KV cache per token = 2 × 64 × 8 × 128 × 2 = 262,144 bytes = 256 KB
For 4096 tokens: 4096 × 256 KB = 1 GB per request

Serving ten concurrent requests at 4096-token contexts on a 70B model requires 10 GB just for the KV cache — before accounting for model weights (~140 GB in BF16). This is why KV cache management determines how many concurrent requests your GPU cluster can handle.

Two Costs: Capacity and Bandwidth

The cache-size arithmetic above frames the KV cache as a capacity problem: how many tokens fit in GPU memory, and therefore how many requests you can serve at once. That is only half the story. The other half — the one that decides per-token latency — is bandwidth.

Autoregressive decoding is memory-bandwidth bound, not compute bound. To generate a single token, the GPU reads every model weight and the entire KV cache out of HBM, then does a comparatively tiny amount of arithmetic: one token’s worth of matrix-vector products. The arithmetic intensity — FLOPs performed per byte moved — is on the order of 1, while a modern accelerator wants hundreds of FLOPs per byte to stay compute-bound. Decode therefore sits deep in the bandwidth-limited region of the roofline, and the clock that matters is HBM bandwidth, not peak TFLOPs.

Decode step, batch size 1, per generated token:

  Read all model weights from HBM      (fixed cost per step)
+ Read the entire KV cache from HBM     (grows with context length)
+ Tiny matrix-vector math for one token
  ───────────────────────────────────
  Time ≈ (weight bytes + KV bytes) / HBM bandwidth

This is the precise sense in which long context is a bandwidth problem. The KV cache must be re-read in full on every decode step, so doubling the context length doubles the KV bytes streamed per token — and in the regime where the cache dominates the weights, that doubles decode latency. No amount of clever packing changes it; PagedAttention removes wasted capacity, but the bytes that hold real tokens still cross the memory bus every step.

A concrete example. An H100 SXM has roughly 3.35 TB/s of HBM3 bandwidth. A Llama 3.1 70B request with a 32K-token context holds 32,768 × 256 KB ≈ 8 GB of KV cache. Reading 8 GB at 3.35 TB/s costs about 2.4 ms — per generated token, just for the cache, before a single weight is touched. At 100K tokens the same request spends over 7 ms per token on KV reads alone, which is why throughput collapses as context grows.

Batching changes the balance but does not escape it. Batching many requests amortises the weight reads — the weights are read once and reused for every sequence in the batch, raising arithmetic intensity for the weight matmuls — which is why large batches are far more efficient per token. But the KV cache is per-sequence: total KV bytes read scale with batch size times context length. Past a certain batch size and context, KV bandwidth, not weight bandwidth, becomes the dominant term, and the only levers left are shrinking the cache (quantization, GQA, MLA) or shortening the effective context. The same memory-bandwidth ceiling that constrains every modern accelerator — see HBM and the memory wall — lands squarely on the KV cache.

The Fragmentation Problem

Before PagedAttention, every major inference system allocated KV cache memory by reserving a contiguous block of GPU memory large enough for the maximum possible output length for each request when the request arrived.

Naive KV cache allocation (pre-PagedAttention):

GPU memory:  [████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░]
              ← reserved for req A (max len) →← reserved for req B →

Actual usage after generation:
  Request A generates 200 tokens, reserved 2048: 1848 tokens wasted
  Request B generates 50 tokens, reserved 2048: 1998 tokens wasted

Memory utilisation: (200 + 50) / (2048 + 2048) = 6%

Two problems arise:

Internal fragmentation: Memory reserved for a request but not yet used. A request reserved for 2048 tokens that has generated 50 tokens wastes 90%+ of its reservation. Because you don’t know how long a request will be when it arrives, you must reserve for the maximum.

External fragmentation: As requests complete and free their reserved blocks, the freed memory is scattered across the GPU address space in non-contiguous chunks. A new large request may not fit even if the total free memory is sufficient — it cannot be assembled from fragments.

These two effects combined wasted 60–80% of available KV cache memory in production serving systems before PagedAttention.

PagedAttention

PagedAttention, introduced by the vLLM team at UC Berkeley in 2023, applies the virtual memory paging concept to KV cache management. Instead of one large contiguous block per request, the cache is divided into fixed-size pages (called blocks), each holding the KV tensors for a fixed number of tokens (the block size, typically 16–32 tokens).

PagedAttention memory layout:

Physical GPU memory blocks (all same size):
┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐
│Blk 0 │ │Blk 1 │ │Blk 2 │ │Blk 3 │ │Blk 4 │ │Blk 5 │ │Blk 6 │
└──────┘ └──────┘ └──────┘ └──────┘ └──────┘ └──────┘ └──────┘

Request A logical KV sequence (200 tokens, block size 16):
  Logical block 0 → Physical block 2
  Logical block 1 → Physical block 0
  Logical block 2 → Physical block 5
  ...
  (block table maps logical → physical)

Request B logical KV sequence (50 tokens):
  Logical block 0 → Physical block 1
  Logical block 1 → Physical block 3
  Logical block 2 → Physical block 4 (partially used)

A block table (stored on the CPU) maps each request’s logical block indices to physical GPU memory blocks. The attention kernel uses this table to locate the correct physical blocks when computing attention.

Key properties:

• No internal fragmentation (almost): Only the last block of each sequence can be partially filled. With a block size of 16 tokens, the maximum waste per request is 15 tokens × token_size.
• No external fragmentation: All blocks are identical size, so any free block can satisfy any request’s next-block allocation.
• Block sharing: Multiple requests can share physical blocks for identical content — this is the mechanism prefix caching builds on.

vLLM reports under 4% memory waste with PagedAttention versus 60–80% with naive allocation. In practice, this translates to 2–4× more concurrent requests at the same memory footprint — which translates directly to 2–4× higher throughput.

Block size trade-off

The block size (tokens per block) affects both memory efficiency and attention kernel performance:

• Small blocks (8–16 tokens): Lower waste in partially-filled last blocks; more block table entries to manage; kernel overhead from many small reads.
• Large blocks (64–128 tokens): Higher waste in last blocks; fewer entries; better kernel efficiency for long sequences.

vLLM defaults to 16 tokens per block, which balances these concerns well for typical workloads.

Continuous Batching

PagedAttention solves where KV blocks live. Continuous batching solves when requests run. The two were co-designed, and neither delivers its full benefit without the other.

The problem it fixes is head-of-line blocking in naive batching. Traditional static batching groups a fixed set of requests, runs them together, and returns no GPU capacity until the entire batch finishes. Because generation lengths vary wildly — one request emits 8 tokens, another 800 — short requests sit idle in the batch waiting for the longest one, and no new request can start until the whole batch drains.

Static batching (batch of 4):

req A ████░░░░░░░░░░░░  done at step 4, then idle
req B ████████████████  done at step 16
req C ██████░░░░░░░░░░  done at step 6, then idle
req D ███░░░░░░░░░░░░░░  done at step 3, then idle
      └─ GPU runs all 16 steps; slots A, C, D wasted after they finish

Continuous batching (iteration-level scheduling):

req A ████▣▣▣▣▣▣▣▣▣▣▣▣  finishes, slot immediately refilled
req B ████████████████
req C ██████▣▣▣▣▣▣▣▣▣▣  finishes, slot refilled
req D ███▣▣▣▣▣▣▣▣▣▣▣▣▣  finishes, slot refilled
      └─ ▣ = a different waiting request admitted mid-flight

Continuous batching — also called iteration-level scheduling or, in TensorRT-LLM, in-flight batching — was introduced by the Orca system (OSDI 2022). The scheduler makes its decision at the granularity of a single decode iteration rather than a whole request. After every step it retires any sequence that just emitted its stop token, frees that sequence’s KV blocks, and admits waiting requests into the freed slots — all without restarting or reallocating the running batch.

This is only practical because of PagedAttention. With contiguous per-request allocation, adding and removing sequences of different lengths from a live batch would constantly fragment memory. With paged blocks, a retiring request simply returns its blocks to the free pool and a new request grabs whatever blocks it needs. vLLM, SGLang, and TensorRT-LLM all combine the two, and the throughput difference over static batching is large — often close to an order of magnitude on workloads with high length variance.

One refinement worth knowing is chunked prefill. Prefill (processing the whole prompt at once) is compute-bound; decode (one token) is memory-bound. If a long prompt’s prefill runs as one monolithic step, it stalls every in-flight decode and produces latency spikes. Chunked prefill (available in vLLM) splits a long prefill into smaller pieces and interleaves them with ongoing decodes, smoothing tail latency at a small cost to raw prefill throughput. For latency-sensitive serving — covered alongside engine choice in the self-hosted inference guide — it is usually worth enabling.

Prefix Caching

Prefix caching extends PagedAttention with a key insight: if two requests share an identical prefix — the same system prompt, the same document context, the same conversation history — the KV blocks for that prefix can be computed once and reused across all requests that share it.

Without prefix caching (3 requests, same system prompt):

Request 1: [System prompt: 500 tokens][User message 1]
  → Compute KV for all 500 system prompt tokens + user message
  → Cost: 500 + len(msg1) tokens of prefill compute

Request 2: [System prompt: 500 tokens][User message 2]
  → Compute KV for all 500 system prompt tokens again
  → Cost: 500 + len(msg2) tokens of prefill compute

Request 3: [System prompt: 500 tokens][User message 3]
  → Compute KV for all 500 system prompt tokens again
  → Cost: 500 + len(msg3) tokens of prefill compute

Total: 3 × 500 = 1500 tokens of redundant computation

With prefix caching:

Request 1: Compute + cache KV blocks for system prompt → Cost: 500 tokens
Request 2: Cache hit on system prompt blocks → Cost: 0 tokens
Request 3: Cache hit on system prompt blocks → Cost: 0 tokens

Total: 500 tokens computed, 1000 tokens saved (66% reduction)

The savings scale with the prefix length as a fraction of total request length. A 500-token system prompt on 600-token average requests (500/600 = 83% prefix ratio) saves 83% of prefill compute on cache-hit requests.

How vLLM implements automatic prefix caching

vLLM computes a hash of the content of each full block. When a new request arrives, vLLM checks whether each block of the prefix has a matching hash in the cache. If it does, the cached physical block is mapped directly into the new request’s block table — no computation required.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

# Enabling automatic prefix caching in vLLM
from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    enable_prefix_caching=True,       # Enable APC
    gpu_memory_utilization=0.90,
    max_model_len=8192,
)

# First request with a long system prompt
sampling_params = SamplingParams(temperature=0.7, max_tokens=256)

system_prompt = "You are a helpful assistant for a DevOps team. " * 100  # long prompt

outputs = llm.generate([
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": "How do I restart a Kubernetes deployment?"}
], sampling_params)

# Second request reuses cached KV blocks for the system prompt
outputs = llm.generate([
    {"role": "system", "content": system_prompt},  # same prefix → cache hit
    {"role": "user", "content": "How do I check pod logs?"}
], sampling_params)

When prefix caching helps

Prefix caching’s value depends entirely on the prefix hit rate — how often incoming requests share a prefix with something already in cache:

For a customer support bot with a 500-token system prompt serving 100 req/s, prefix caching may eliminate 85% of prefill compute. For a diverse single-shot API with no shared context, it helps almost nothing.

SGLang RadixAttention

SGLang’s RadixAttention takes prefix caching further by using a radix tree (trie) structure to cache KV at arbitrary prefix lengths, not just full blocks. This handles cases where prefixes don’t align to block boundaries and produces higher cache utilisation for varied-length shared prefixes.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# SGLang with RadixAttention (enabled by default)
import sglang as sgl

@sgl.function
def multi_turn_chat(s, system_prompt, messages):
    s += sgl.system(system_prompt)
    for user_msg, _ in messages[:-1]:
        s += sgl.user(user_msg)
        s += sgl.assistant(sgl.gen("response"))
    s += sgl.user(messages[-1][0])
    s += sgl.assistant(sgl.gen("final_response", max_new_tokens=512))

SGLang automatically caches and reuses KV for any shared prefix, including partial blocks. In benchmarks on multi-turn workloads, SGLang’s RadixAttention produces 20–40% higher cache hit rates than vLLM’s hash-based approach.

Cross-request KV sharing with LMCache

LMCache extends prefix caching beyond a single serving instance. It provides a distributed KV cache layer that stores evicted GPU KV blocks on CPU DRAM or SSDs and shares them across multiple inference engine instances:

┌─────────────────────────────────────────────────────────────┐
│                    LMCache Architecture                      │
│                                                              │
│  Request → Router (prefix-aware) → vLLM instance 1          │
│                                  → vLLM instance 2          │
│                                  → vLLM instance 3          │
│                                          │                  │
│                                          ▼                  │
│                              ┌──────────────────────┐       │
│                              │  LMCache KV store     │       │
│                              │  (GPU → CPU → SSD)   │       │
│                              │  Shared across nodes  │       │
│                              └──────────────────────┘       │
└─────────────────────────────────────────────────────────────┘

For enterprise deployments with multiple serving replicas all handling requests with the same large document context, LMCache can produce dramatic reductions in total prefill compute — one instance computes the KV blocks, all others reuse them.

Quantized KV Cache

Even with PagedAttention eliminating fragmentation and prefix caching eliminating redundant computation, the KV cache itself can be large. Quantizing the cached K and V tensors from BF16 (2 bytes per value) to FP8 or INT8 (1 byte per value) halves the memory footprint per cached token.

FP8 KV cache

FP8 (8-bit floating point, E4M3 format) is now hardware-accelerated on NVIDIA Hopper (H100) and Ampere (A100) GPUs. vLLM added FP8 KV cache quantization in 2024 with a per-tensor or per-head scaling factor:

1
2
3
4
5
6
7

# vLLM FP8 KV cache
llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    kv_cache_dtype="fp8",             # Quantize KV cache to FP8
    gpu_memory_utilization=0.90,
    enable_prefix_caching=True,
)

The FP8 cache quantizes K and V using a learned or dynamic scale factor, stores them as FP8, and dequantizes on read during attention computation. The dequantization adds a small overhead, but the halved memory footprint allows approximately double the batch size or context length for the same GPU memory — typically producing net throughput improvements of 20–40% in memory-bound serving scenarios.

Accuracy impact: vLLM’s April 2026 blog post on FP8 KV cache reports minimal degradation for most benchmarks — typically under 0.5% on standard NLP benchmarks for E4M3 quantization. Outlier tokens (tokens with very large K or V values) can cause larger errors; per-head scaling (rather than per-tensor) mitigates this.

INT8 KV cache

INT8 quantization uses symmetric or asymmetric integer quantization with a scale and zero-point. It is more widely supported on older hardware (Turing, Ampere without FP8 support) but requires more careful calibration to avoid accuracy degradation from outliers.

1
2
3
4
5
6

# vLLM INT8 KV cache
llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    kv_cache_dtype="int8",
    gpu_memory_utilization=0.90,
)

Per recent GPU-accelerated INT8 quantization research, 8× compression (equivalent to 2-bit effective representation via block quantization) retains over 90% of full-cache performance on most tasks. For most serving applications, INT8 KV cache is a safe default when FP8 is unavailable.

Mixed-precision KV cache

Recent work (KVmix, 2025) shows that layers contribute differently to model quality and can tolerate different quantization levels. Early and late layers are more sensitive; middle layers tolerate more aggressive quantization. Mixed-precision KV caching applies 4-bit or 2-bit quantization to tolerant layers and keeps FP8 or BF16 for sensitive ones:

1
2
3
4
5
6
7

# Conceptual mixed-precision KV cache (research; not yet standard in vLLM)
# Layers 0-7 and 24-31: FP8 (more sensitive)
# Layers 8-23: INT4 (tolerant)
kv_cache_config = {
    "sensitive_layers": {"dtype": "fp8", "layers": list(range(0,8)) + list(range(24,32))},
    "tolerant_layers": {"dtype": "int4", "layers": list(range(8,24))},
}

This approach can achieve 4–6× overall cache compression while maintaining quality close to FP16/BF16 baseline.

Attention Variants: Shrinking the Cache at the Source

Every optimisation so far operates on a cache whose size is fixed by the model architecture. But the architecture itself decides how many K and V tensors exist per token — and that is the most fundamental lever of all, because it shrinks the cache before any serving trick runs.

Recall the per-token cache formula: 2 × n_layers × n_kv_heads × head_dim × dtype_bytes. The n_kv_heads term is set by the attention design, and four variants have emerged, each trading quality for cache size differently.

Multi-Head Attention (MHA) is the original design from the 2017 Transformer: every one of the H query heads has its own K and V head, so n_kv_heads = n_heads. Maximum quality, maximum cache.

Multi-Query Attention (MQA) collapses all K and V to a single shared head — n_kv_heads = 1 — while keeping all H query heads. For a 32-head model that is a 32× reduction in KV cache. The cost is real: quality degrades and training can become unstable, because all query heads must agree on one key/value subspace.

Grouped-Query Attention (GQA) is the pragmatic middle ground that almost every current open model uses. Query heads are partitioned into G groups, each group sharing one K/V head, so n_kv_heads = G. Llama 3 (8 KV heads for 32 query heads), Mistral, and Gemma 2 use GQA at roughly a 4–8× cache reduction while recovering nearly all of MHA’s quality. GQA can even be distilled from an existing MHA checkpoint rather than trained from scratch.

Multi-head Latent Attention (MLA), introduced with DeepSeek-V2, takes a different route: instead of storing per-head K and V, it projects them down into a single low-rank latent vector that is cached, then projects back up on the fly during attention. What lands in the cache is one small latent per token rather than K and V for every head, which DeepSeek reports as a greater-than-90% KV reduction versus MHA — with quality matching or exceeding GQA. The cost moves into extra projection compute and a more intricate kernel.

The practical takeaway when selecting a model for constrained hardware or very long context: the attention variant outweighs cache quantization. A GQA model and an MHA model both quantized to FP8 still differ by the same 4–8× in KV footprint they did at BF16 — the architectural reduction and the quantization reduction multiply. For long-context serving on a memory budget, prefer GQA or MLA models, then quantize on top.

Memory Layout: Everything Together

Combining the three optimisations changes the effective capacity of a GPU for LLM serving dramatically:

Example: 80GB A100, serving Llama 3.1 8B

Model weights (BF16):  ~16 GB
KV cache budget:       ~60 GB (after weights + framework overhead)

Without any optimisation:
  KV cache per token: 128 KB (BF16, 32 layers, 8 heads, 128 dim)
  Naive allocation (60% waste): effective 24 GB for KV
  Tokens in cache: 24 GB / 128 KB ≈ 187,500 tokens
  At 2048-token avg context: ~91 concurrent requests

With PagedAttention (< 4% waste):
  Effective 57.6 GB for KV
  Tokens in cache: 57.6 GB / 128 KB ≈ 450,000 tokens
  At 2048 tokens: ~219 concurrent requests  (+140%)

With PagedAttention + FP8 KV cache:
  KV cache per token: 64 KB (half of BF16)
  Tokens in cache: 57.6 GB / 64 KB ≈ 900,000 tokens
  At 2048 tokens: ~439 concurrent requests  (+382% vs baseline)

With prefix caching on top (80% hit rate, 500-token shared prefix):
  Each request's effective compute: 20% of prefill
  Throughput multiplier on prefill: ~5×
  Combined effect: dramatically higher QPS

These multipliers compound. PagedAttention + FP8 + prefix caching together can improve effective serving capacity by 5–10× compared to naive approaches, for the same hardware.

Practical Configuration

vLLM production configuration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",

    # Parallelism
    tensor_parallel_size=4,            # Split across 4 GPUs

    # Memory
    gpu_memory_utilization=0.90,       # Leave 10% headroom
    max_model_len=8192,                # Max context window

    # KV cache optimisations
    kv_cache_dtype="fp8",              # Halve KV memory
    enable_prefix_caching=True,        # Cache shared prefixes

    # Block size (tune per workload)
    block_size=16,                     # Default; try 32 for long contexts
)

Monitoring cache utilisation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# vLLM exposes metrics via Prometheus
# Key metrics to watch:

# KV cache utilisation (fraction of blocks in use)
vllm:cache_config_info{...}
vllm:gpu_cache_usage_perc

# Prefix cache hit rate
vllm:prefix_cache_hit_rate

# Request queue depth (sign of insufficient KV capacity)
vllm:num_requests_waiting

A healthy serving deployment shows:

• gpu_cache_usage_perc: 60–85% (headroom for burst without OOM)
• prefix_cache_hit_rate: >50% for chatbot/RAG workloads
• num_requests_waiting: near zero at steady state

If gpu_cache_usage_perc is consistently above 90%, you are memory-bound: reduce max_model_len, enable FP8 KV cache, add a GPU, or reduce batch size.

If prefix_cache_hit_rate is under 20% but your application has shared prefixes, investigate whether requests are arriving in a consistent prefix order. Prefix caching requires the prefix to be in cache — if requests with the same system prompt are spread across multiple serving replicas without shared cache state (no LMCache), each replica’s hit rate is independent and lower.

Tuning for different workloads

What Matters Most

The leverage order for KV cache engineering:

1. PagedAttention: Already the default in vLLM, SGLang, and TensorRT-LLM. If you’re on a framework that doesn’t use it, switch.

2. Continuous batching: The scheduler that keeps the GPU saturated by admitting and retiring requests at every decode iteration. Inseparable from PagedAttention in every modern engine; together they are the single largest throughput win over naive static batching.

3. Prefix caching: Highest-impact optimisation for any workload with shared context. Enable it unconditionally — it costs nothing when the hit rate is zero and saves significantly when it isn’t.

4. FP8 KV cache: 2× memory reduction with minimal accuracy cost on Hopper/Ampere hardware. Enable for any memory-constrained serving scenario.

5. Attention architecture (GQA/MLA): A model-level decision that shrinks the cache before any serving trick runs. GQA (Llama 3, Mistral, Gemma 2) cuts KV heads 4–8×; MLA (DeepSeek-V2/V3) compresses K and V to a low-rank latent and goes further still. When choosing models for constrained hardware or long context, the attention variant matters more than any cache quantization, and the two reductions multiply.

6. INT4/mixed-precision cache: Emerging, not yet mainstream in production frameworks. Worth watching for memory-critical deployments in 2026.

Verdict

Naive KV cache handling wastes most of a GPU and caps you at a fraction of the requests the hardware could serve. The modern stack fixes that in layers: PagedAttention removes fragmentation, continuous batching keeps every iteration full, prefix caching deletes redundant prefill, and FP8/INT8 plus GQA or MLA shrink each cached token. Stacked, these turn into a 5–10× swing in effective serving capacity on the same silicon.

But the deeper lesson is architectural. Once fragmentation is gone, long-context inference stops being a capacity problem and becomes a bandwidth problem: every decode step must stream the entire cache out of HBM, so doubling the context roughly doubles per-token decode cost no matter how cleverly the memory is packed. That is why the next round of gains is coming from attention variants that shrink the cache at the source rather than from compressing it after the fact. Optimise capacity and you serve more users; optimise bandwidth and you serve them faster. Production systems need both.

Sources

• Efficient Memory Management with PagedAttention — arXiv 2309.06180
• Orca: A Distributed Serving System for Transformer-Based Generative Models — OSDI 2022
• How Continuous Batching Enables 23× Throughput — Anyscale
• Automatic Prefix Caching — vLLM Docs
• The State of FP8 KV-Cache in vLLM — vLLM Blog
• Quantized KV Cache — vLLM Docs
• Fast Transformer Decoding: One Write-Head is All You Need (MQA) — arXiv 1911.02150
• GQA: Training Generalized Multi-Query Transformer Models — arXiv 2305.13245
• DeepSeek-V2 and Multi-head Latent Attention — arXiv 2405.04434
• LMCache: KV Cache Layer for Enterprise LLM Inference — arXiv
• Prefix Caching — BentoML LLM Inference Handbook
• How PagedAttention Resolves Memory Waste — Red Hat Developer