The KV cache is the single biggest lever for LLM inference cost and throughput. How it works, why naive allocation wastes 60–80% of GPU memory, how PagedAttention and continuous batching fix that, what prefix caching actually saves, how FP8/INT8 quantization and GQA/MLA shrink the cache, and why long context is fundamentally a memory-bandwidth problem.
Inference
-
KV Cache Engineering: PagedAttention, Continuous Batching, Attention Variants, and the Bandwidth Wall -
LLM Quantization Compared: GGUF, AWQ, GPTQ, FP8, and the int4 Cliff A practical comparison of the quantization formats that decide whether a model fits your GPU: weight-only versus activation quantization, what calibration data actually buys you, the accuracy cost of int4, and which of GGUF, GPTQ, AWQ, and FP8 makes sense for which workload.
-
Speculative Decoding Explained: Trading Compute for Bandwidth to Decode Faster How draft-model speculative sampling makes LLM inference 2-3x faster without changing the output: why decode is bandwidth-bound, the draft-verify-accept loop, the rejection-sampling step that keeps it lossless, the acceptance-rate math that decides the speedup, EAGLE and Medusa as learned drafters, and the batch-size regime where speculation quietly makes things slower.
-
Bayesian Statistics for Engineers Bayesian and frequentist statistics answer different questions, and engineers benefit from knowing which question they are actually asking. This post walks through priors, conjugate posteriors, a worked Bayesian A/B test, MCMC at a working level, and where Bayesian reasoning genuinely beats classical inference.
-
Mixture of Experts, Honestly: Why Every Frontier Model Went Sparse and What It Actually Costs The honest accounting on Mixture of Experts: why every frontier model in 2026 is sparse, what the "X total / Y active" parameter math actually buys you, the routing problems the marketing skips, and the all-to-all communication tax that decides whether a MoE model ships or stalls.
-
DiffusionGemma: When the LLM Stops Writing Left to Right DiffusionGemma is Google's first open-weight diffusion language model — text generated by denoising blocks in parallel instead of one token at a time. How it differs from autoregressive Gemma, the real speed win and the honest benchmark tax, and exactly what it takes to run the thing on your own GPU.
-
Constrained Generation: Outlines, JSON Mode, and Structured Output That Works How regex-constrained sampling and grammar-guided decoding actually work at the token level, why prompt-only JSON mode fails 5–20% of the time in production, and the full toolchain for guaranteed structured output: Outlines, XGrammar, vLLM guided decoding, and Instructor.
-
Edge AI Accelerators: Coral, Hailo, and Jetson Orin Nano Super Compared A practical guide to edge AI accelerators for the homelab: Google Coral TPU, Hailo-8 and Hailo-8L, and the Jetson Orin Nano Super. Real benchmarks, power draw, use case fit, and where each beats or loses to a small discrete GPU.
-
Mixture of Experts Internals: Routing, Expert Parallelism, and Load Balancing MoE is now the default architecture for top-tier open models. This is how it actually works: router design, top-k token dispatch, auxiliary loss for load balancing, expert parallelism across GPUs, and what it means for inference serving — with DeepSeek-V3 and Mixtral as concrete case studies.
-
Quantization Deep Dive: GPTQ, AWQ, GGUF, AQLM, and MLX A rigorous look at every major LLM quantization format: how GPTQ's Hessian-guided rounding works, why AWQ's activation-aware scaling beats naive per-channel approaches, when GGUF k-quants are the right choice, where AQLM wins at sub-3-bit, and how MLX handles Apple Silicon. Includes conversion workflows, a perplexity shootout, and a decision guide.
-
Speculative Decoding: Draft Models, EAGLE, and How to Actually Use It How speculative decoding achieves 2–4x inference speedup without changing model outputs, covering the rejection-sampling proof, EAGLE and EAGLE2 draft strategies, ngram lookahead, and production configuration for vLLM and llama.cpp.
-
Triton Inference Server: Production ML Serving That Actually Scales NVIDIA Triton Inference Server for production ML serving — dynamic batching, concurrent model execution, multiple backends, and the GPU-utilization and model-management problems it solves between a trained model and serving at scale.
-
Self-Hosted AI Inference: vLLM, llama.cpp, and Running Your Own OpenAI-Compatible API A comprehensive guide to running your own LLM inference servers — covering vLLM, llama.cpp, OpenAI-compatible APIs, batching strategies, quantization, and benchmarking throughput so you can make informed hardware and software decisions.