The KV cache is the single biggest lever for LLM inference cost and throughput. How it works, why naive allocation wastes 60–80% of GPU memory, how PagedAttention and continuous batching fix that, what prefix caching actually saves, how FP8/INT8 quantization and GQA/MLA shrink the cache, and why long context is fundamentally a memory-bandwidth problem.
Memory-Bandwidth
-
KV Cache Engineering: PagedAttention, Continuous Batching, Attention Variants, and the Bandwidth Wall -
HBM4 and the Memory Wall for AI Why every modern AI accelerator is pin-limited on bandwidth rather than starved for math, how High Bandwidth Memory actually works, what each generation from HBM2 to HBM4 buys, why LLM inference is a memory problem in disguise, and the three-company supply story that makes HBM the real binding constraint on AI economics.