A rigorous look at every major LLM quantization format: how GPTQ's Hessian-guided rounding works, why AWQ's activation-aware scaling beats naive per-channel approaches, when GGUF k-quants are the right choice, where AQLM wins at sub-3-bit, and how MLX handles Apple Silicon. Includes conversion workflows, a perplexity shootout, and a decision guide.
Llama.cpp
-
Quantization Deep Dive: GPTQ, AWQ, GGUF, AQLM, and MLX -
Speculative Decoding: Draft Models, EAGLE, and How to Actually Use It How speculative decoding achieves 2–4x inference speedup without changing model outputs, covering the rejection-sampling proof, EAGLE and EAGLE2 draft strategies, ngram lookahead, and production configuration for vLLM and llama.cpp.
-
Llama 4 and Gemma 4: The 2026 Self-Hosted Model Landscape Meta shipped Llama 4 Scout and Maverick and Google shipped four Gemma 4 variants under Apache 2.0 — all within a few weeks of each other. This is a practical walkthrough of what each model is, what hardware you actually need, how to deploy them with Ollama, vLLM, and llama.cpp, and which one to pick for which job.
-
Self-Hosted AI Inference: vLLM, llama.cpp, and Running Your Own OpenAI-Compatible API A comprehensive guide to running your own LLM inference servers — covering vLLM, llama.cpp, OpenAI-compatible APIs, batching strategies, quantization, and benchmarking throughput so you can make informed hardware and software decisions.
-
Local LLM Inference for Coding: The Complete 2025/2026 Guide Everything you need to run powerful AI coding assistants locally — model benchmarks, Ollama, LM Studio, llama.cpp, hardware requirements, and editor integration with Continue.dev and Aider.