The KV cache is the single biggest lever for LLM inference cost and throughput. How it works, why naive allocation wastes 60–80% of GPU memory, how PagedAttention and continuous batching fix that, what prefix caching actually saves, how FP8/INT8 quantization and GQA/MLA shrink the cache, and why long context is fundamentally a memory-bandwidth problem.
Vllm
-
KV Cache Engineering: PagedAttention, Continuous Batching, Attention Variants, and the Bandwidth Wall -
LLM Quantization Compared: GGUF, AWQ, GPTQ, FP8, and the int4 Cliff A practical comparison of the quantization formats that decide whether a model fits your GPU: weight-only versus activation quantization, what calibration data actually buys you, the accuracy cost of int4, and which of GGUF, GPTQ, AWQ, and FP8 makes sense for which workload.
-
Speculative Decoding Explained: Trading Compute for Bandwidth to Decode Faster How draft-model speculative sampling makes LLM inference 2-3x faster without changing the output: why decode is bandwidth-bound, the draft-verify-accept loop, the rejection-sampling step that keeps it lossless, the acceptance-rate math that decides the speedup, EAGLE and Medusa as learned drafters, and the batch-size regime where speculation quietly makes things slower.
-
Constrained Generation: Outlines, JSON Mode, and Structured Output That Works How regex-constrained sampling and grammar-guided decoding actually work at the token level, why prompt-only JSON mode fails 5–20% of the time in production, and the full toolchain for guaranteed structured output: Outlines, XGrammar, vLLM guided decoding, and Instructor.
-
Quantization Deep Dive: GPTQ, AWQ, GGUF, AQLM, and MLX A rigorous look at every major LLM quantization format: how GPTQ's Hessian-guided rounding works, why AWQ's activation-aware scaling beats naive per-channel approaches, when GGUF k-quants are the right choice, where AQLM wins at sub-3-bit, and how MLX handles Apple Silicon. Includes conversion workflows, a perplexity shootout, and a decision guide.
-
Speculative Decoding: Draft Models, EAGLE, and How to Actually Use It How speculative decoding achieves 2–4x inference speedup without changing model outputs, covering the rejection-sampling proof, EAGLE and EAGLE2 draft strategies, ngram lookahead, and production configuration for vLLM and llama.cpp.
-
Llama 4 and Gemma 4: The 2026 Self-Hosted Model Landscape Meta shipped Llama 4 Scout and Maverick and Google shipped four Gemma 4 variants under Apache 2.0 — all within a few weeks of each other. This is a practical walkthrough of what each model is, what hardware you actually need, how to deploy them with Ollama, vLLM, and llama.cpp, and which one to pick for which job.
-
Self-Hosted AI Inference: vLLM, llama.cpp, and Running Your Own OpenAI-Compatible API A comprehensive guide to running your own LLM inference servers — covering vLLM, llama.cpp, OpenAI-compatible APIs, batching strategies, quantization, and benchmarking throughput so you can make informed hardware and software decisions.