A practical comparison of the quantization formats that decide whether a model fits your GPU: weight-only versus activation quantization, what calibration data actually buys you, the accuracy cost of int4, and which of GGUF, GPTQ, AWQ, and FP8 makes sense for which workload.
Llama-Cpp
-
LLM Quantization Compared: GGUF, AWQ, GPTQ, FP8, and the int4 Cliff -
Jan: The Open-Source Local LLM Runner Coming for Ollama and LM Studio Jan bundles a chat UI, a model hub, and an OpenAI-compatible server on top of a Cortex/llama.cpp engine — Apache 2.0, no account, no telemetry. A hands-on look at where it beats Ollama and LM Studio, where it doesn't, and how to tune it.