The KV cache is the single biggest lever for LLM inference cost and throughput. How it works, why naive allocation wastes 60–80% of GPU memory, how PagedAttention and continuous batching fix that, what prefix caching actually saves, how FP8/INT8 quantization and GQA/MLA shrink the cache, and why long context is fundamentally a memory-bandwidth problem.
Mlops
-
KV Cache Engineering: PagedAttention, Continuous Batching, Attention Variants, and the Bandwidth Wall -
LLM Quantization Compared: GGUF, AWQ, GPTQ, FP8, and the int4 Cliff A practical comparison of the quantization formats that decide whether a model fits your GPU: weight-only versus activation quantization, what calibration data actually buys you, the accuracy cost of int4, and which of GGUF, GPTQ, AWQ, and FP8 makes sense for which workload.
-
Speculative Decoding Explained: Trading Compute for Bandwidth to Decode Faster How draft-model speculative sampling makes LLM inference 2-3x faster without changing the output: why decode is bandwidth-bound, the draft-verify-accept loop, the rejection-sampling step that keeps it lossless, the acceptance-rate math that decides the speedup, EAGLE and Medusa as learned drafters, and the batch-size regime where speculation quietly makes things slower.
-
LLM Evals: Testing Your AI Application Like Real Software Writing evaluations that catch regressions before they reach users: golden datasets, LLM-as-judge pitfalls, CI integration, and the tools that make a sustainable eval pipeline — Braintrust, Langfuse, Promptfoo, Inspect, and DeepEval compared.
-
Triton Inference Server: Production ML Serving That Actually Scales NVIDIA Triton Inference Server for production ML serving — dynamic batching, concurrent model execution, multiple backends, and the GPU-utilization and model-management problems it solves between a trained model and serving at scale.
-
GPU Infrastructure for ML: CUDA, MIG, Kubernetes Device Plugins, and Cost-Efficient Training A practical guide to GPU infrastructure for machine learning — CUDA fundamentals, NVIDIA MIG for multi-tenancy, sharing GPUs in Kubernetes with device plugins and time-slicing, building cost-efficient training clusters, and monitoring GPU utilization.
-
MLOps Fundamentals: Experiment Tracking, Model Registry, Serving, and Drift Monitoring A practical guide to MLOps — structuring experiments with MLflow, managing the model lifecycle through a registry, serving models in production with BentoML and Triton Inference Server, and detecting data and concept drift before it silently degrades your models.
-
Self-Hosted AI Inference: vLLM, llama.cpp, and Running Your Own OpenAI-Compatible API A comprehensive guide to running your own LLM inference servers — covering vLLM, llama.cpp, OpenAI-compatible APIs, batching strategies, quantization, and benchmarking throughput so you can make informed hardware and software decisions.
-
Fine-Tuning Small Models: When and Why to Fine-Tune vs. Prompt Engineer A practical guide to fine-tuning small language models — understanding when fine-tuning beats prompt engineering, how LoRA and QLoRA work, training a model on your own data with Unsloth or Axolotl, and deploying the result with Ollama.