LLM Quantization Compared: GGUF, AWQ, GPTQ, FP8, and the int4 Cliff

A 70-billion-parameter model in 16-bit weights needs 140 GB just to hold the parameters, before a single token of context. Quantize those weights to 4 bits and the same model fits in roughly 40 GB — one card instead of two, a consumer GPU instead of a datacenter one, a model that runs at all instead of one that swaps to disk and dies. That is why quantization is not an optimization you reach for late; for anyone running models outside a hyperscaler it is the thing that decides what you can run at all. The catch is that “4-bit” is not one technique. GGUF, GPTQ, AWQ, and FP8 make different bets about what to compress, how to choose the rounding, and where the quality goes when it goes. This is what each one actually does and how to pick.

Why Quantization Works At All

The reason you can throw away most of the bits in a weight and keep most of the model’s behavior comes down to two facts. First, neural network weights are highly redundant and roughly normally distributed around zero; the information that matters is in the relative magnitudes, not in 16 bits of mantissa precision. Second — and this is the part that makes quantization a throughput win, not just a memory win — autoregressive decoding is memory-bandwidth bound, not compute bound.

During token generation you stream the entire weight matrix through the GPU’s compute units to produce one token, then do it again for the next token. The arithmetic per weight is trivial; the bottleneck is moving the weights from VRAM (or worse, from system RAM) into the registers. Halving the bytes per weight roughly halves the bytes you move, which roughly doubles decode throughput on a bandwidth-bound model. So quantization buys you two things at once: the model fits, and it decodes faster. This is the same bandwidth story behind KV cache optimization and speculative decoding — at inference time, bytes moved is the currency that matters.

The cost is precision. The entire game is choosing a mapping from high-precision floats to low-bit integers (or low-bit floats) that loses as little of the model’s behavior as possible.

The Mechanics: Scales, Zero-Points, and Groups

Integer quantization maps a block of floating-point weights onto a small set of integer levels using a scale and (sometimes) a zero-point:

  FP16 weights in one group (e.g. 128 values)
  -0.91  ...  +0.88        <- real range [w_min, w_max]
     |          |
     v          v
  scale = (w_max - w_min) / (2^bits - 1)
  q = round(w / scale) + zero_point      <- store q (4 bits) + scale (16 bits)

  At inference, dequantize on the fly:
  w_hat = (q - zero_point) * scale        <- never perfectly equal to w

   16-bit weight:  [ s | eeeee | mmmmmmmmmm ]   2 bytes
   4-bit weight:   [ qqqq ]  + shared scale     ~0.5 byte + overhead

The key knob is group size: how many weights share one scale. A single scale for an entire tensor (per-tensor) is cheap to store but crude — one outlier weight stretches the range and wastes precision on everyone else. A scale per small group (commonly 128 or 64 weights) tracks the local distribution far better, at the cost of storing more scales. Most modern 4-bit schemes use group sizes of 32–128, which is why a “4-bit” model is really closer to 4.5 bits per weight once you count the scales. That overhead is the difference between a quant that holds together and one that falls apart.

The hard part is outliers. In large transformers, a handful of activation channels carry values 10–100x larger than the rest, and naive quantization spends its entire dynamic range trying to represent them, crushing everything else to zero. Every serious method below is, at heart, a different strategy for handling outliers.

Weight-Only vs Weight-and-Activation

The single most important distinction in this whole space is what you quantize:

• Weight-only quantization compresses the stored weights to 4 or 3 bits, then dequantizes them back to FP16 inside the GPU before the matrix multiply. The math runs in FP16; you save memory and bandwidth but not arithmetic. GGUF, GPTQ, AWQ, and bitsandbytes NF4 are all weight-only.
• Weight-and-activation quantization quantizes both the weights and the activations flowing through the network, so the matrix multiply itself runs in low precision (int8 or FP8) on the GPU’s tensor cores. This saves arithmetic too, which matters for compute-bound regimes like prefill and large-batch serving. FP8 and int8 SmoothQuant are the common examples.

The reason weight-only dominates for local and small-scale inference is that decode is bandwidth-bound, so saving arithmetic buys little, and quantizing activations is harder — activations have those nasty outliers and change with every input, so they can’t be calibrated as thoroughly as static weights. The reason FP8 dominates at datacenter scale is that high-throughput serving with big batches is compute-bound, and there the FP8 tensor-core speedup on Hopper and Blackwell GPUs is real.

                weight bits   activation   matmul runs in   best for
weight-only     4 (or 3)      FP16         FP16             single-stream, fits-on-one-GPU
FP8 (W8A8)      8 (float)     FP8          FP8 tensor core  high-throughput serving
int8 SmoothQuant 8            int8         int8 tensor core compute-bound, older HW

The Formats, One at a Time

GGUF (llama.cpp)

GGUF is the file format and quantization scheme of llama.cpp, and it is what almost everyone running models locally on mixed hardware actually uses. Its defining feature is k-quants: mixed-precision blocks where more important weights (and the scales themselves) get more bits. A Q4_K_M model, the popular default, uses 4-bit weights for most tensors but bumps the attention and feed-forward layers that matter most to higher precision, landing around 4.8 bits per weight on average.

GGUF’s superpower is CPU/GPU hybrid execution: it can offload as many layers as fit to the GPU and run the rest on CPU, so a model that doesn’t fit in VRAM still runs (slowly). It also supports an importance matrix (imatrix) — calibration data that weights the quantization error by how much each weight actually affects outputs, measurably improving low-bit quants. The trade-off: GGUF is built for llama.cpp/Ollama, not for high-throughput batched serving, where vLLM-native formats win.

1
2
3
4
5

# Quantize an FP16 GGUF to Q4_K_M, optionally with an importance matrix
./llama-quantize --imatrix imatrix.dat model-f16.gguf model-Q4_K_M.gguf Q4_K_M

# Run it, offloading 35 layers to the GPU
./llama-cli -m model-Q4_K_M.gguf -ngl 35 -p "Explain quantization in one paragraph"

GPTQ

GPTQ is a one-shot, post-training method that quantizes weights layer by layer while using second-order (Hessian) information from a small calibration set to compensate: as it rounds each weight, it adjusts the not-yet-quantized weights in the same layer to cancel the error it just introduced. The result is strong 4-bit and even usable 3-bit accuracy. GPTQ needs a calibration dataset (typically 128 samples of representative text) and a non-trivial quantization pass, but the output is a compact, fast-loading model with mature kernel support in vLLM, TGI, and ExLlama.

1
2
3
4
5
6
7

from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig

tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
cfg = GPTQConfig(bits=4, group_size=128, dataset="c4", tokenizer=tok)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct", quantization_config=cfg, device_map="auto")
model.save_pretrained("llama-3.1-8b-gptq-4bit")

AWQ (Activation-aware Weight Quantization)

AWQ starts from an observation: not all weights matter equally, and the ones that matter most are the ones multiplied by large-magnitude activations. Rather than touch those salient weight channels (about 1%), AWQ scales them up before quantizing and scales the activations down to match, protecting the important channels from rounding error without any mixed precision or backprop. It needs only a small calibration set to find the salient channels, runs fast, and tends to match or beat GPTQ on instruction-tuned models while being more robust to the choice of calibration data. AWQ 4-bit is a very common default for vLLM serving.

1
2
3
4
5
6
7
8
9

pip install autoawq
python -c "
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
m = AutoAWQForCausalLM.from_pretrained('meta-llama/Llama-3.1-8B-Instruct')
t = AutoTokenizer.from_pretrained('meta-llama/Llama-3.1-8B-Instruct')
m.quantize(t, quant_config={'w_bit':4,'q_group_size':128,'zero_point':True})
m.save_quantized('llama-3.1-8b-awq')
"

FP8

FP8 is the datacenter answer. Instead of integers, it uses an 8-bit floating-point format — usually E4M3 (4 exponent, 3 mantissa bits) for weights and activations, E5M2 (more range, less precision) where dynamic range matters. Because it keeps an exponent, FP8 handles outliers far more gracefully than int8, which is why W8A8 FP8 often runs at near-BF16 accuracy with almost no calibration fuss. On Hopper (H100) and Blackwell GPUs the tensor cores execute FP8 matmuls natively at roughly double the FP16 rate, so FP8 is the rare quant that helps both memory and compute. The cost: it needs that hardware, and at 8 bits it saves only half the memory of FP16, not the three-quarters that int4 saves.

1
2

# vLLM can quantize to FP8 on the fly at load time (Hopper/Blackwell)
vllm serve meta-llama/Llama-3.1-70B-Instruct --quantization fp8 --tensor-parallel-size 2

bitsandbytes (NF4 / int8)

bitsandbytes is the Hugging Face default for zero-effort quantization: load_in_4bit=True and you get NF4 (a 4-bit “normal float” tuned for normally-distributed weights), no calibration step at all. It is the backbone of QLoRA fine-tuning and the fastest way to get a big model onto a small GPU. The trade-off is inference speed: its kernels are slower than GPTQ/AWQ for pure serving, so it shines for experimentation and training, less so for production throughput.

What It Costs: The Accuracy Picture

Here is the honest summary, with the caveat that exact numbers vary by model, task, and benchmark. Perplexity degradation and downstream-task accuracy loss both grow as bits shrink, and the curve has a knee.

Three patterns are worth internalizing. 8-bit is nearly free. FP8 and int8 land so close to FP16 that for most workloads the only question is whether your hardware accelerates them. 4-bit with a good method is the sweet spot. AWQ, GPTQ, and Q4_K_M all sit on the gentle part of the curve — a small, usually acceptable quality hit for a ~3.5x memory reduction. Below 4 bits is a cliff, and it is steeper for small models. A 7B model at 3-bit degrades visibly; a 70B model at 3-bit is often fine because the larger model has more redundancy to spare. This is the single most useful rule of thumb: prefer a bigger model at lower bits over a smaller model at higher bits, down to about 4 bits. A 70B-Q4 will usually beat a 13B-Q8 that uses the same VRAM.

Where Quantization Does Not Help (Or Hurts)

Quantization is not free throughput everywhere. The dequantization step — turning 4-bit weights back into FP16 before the matmul — costs cycles, and that cost is fixed per weight regardless of batch size. In large-batch serving, where you are already compute-bound because many requests share each weight load, weight-only 4-bit quantization can actually be slower than FP16, because you pay dequant overhead without the bandwidth savings paying off. This is exactly the regime where FP8 wins instead: it accelerates the matmul itself.

Other honest caveats:

• Prefill is compute-bound. Weight-only quantization speeds up decode, not the prompt-processing phase. Long-prompt, short-output workloads see less benefit than chatty short-prompt ones.
• The KV cache is a separate problem. Quantizing weights does nothing for KV cache memory, which dominates at long context. KV cache quantization (FP8 or int8 KV) is a distinct knob with its own accuracy trade-offs.
• Calibration data leaks into behavior. GPTQ and AWQ tune the quantization to a calibration set; a mismatched set (e.g. English-only calibration for a multilingual deployment, or generic web text for a code model) can quietly degrade the cases you care about. Use calibration data that resembles your real traffic.
• Quality loss hides from perplexity. A quant can post nearly identical perplexity yet fail more on long-form reasoning, code correctness, or rare-language tasks. Always evaluate on something close to your workload, not just a perplexity number from a blog.

Choosing For Your Workload

The decision collapses to a few questions. Are you bandwidth-bound (single stream, fits on one GPU) or compute-bound (big batch, datacenter)? If bandwidth-bound, weight-only 4-bit; if compute-bound, FP8 or int8. Is your hardware Hopper-or-newer? If yes, FP8 is the easy default for serving. Is the model big (≥30B)? Then you can push below 4 bits; if it is small (≤13B), stay at 4-bit or above. And always: quantize, then measure on your own evals before trusting it. For a fuller treatment of running these locally, see the local LLM coding guide; for whether to self-host at all, the self-hosting vs cloud trade-off applies directly.

Verdict

Quantization is the lever that turns “I need a datacenter” into “this runs on my desk,” and in 2026 the choices have settled into clear lanes. For local and single-GPU use, weight-only 4-bit is the default that just works: GGUF Q4_K_M if your hardware is mixed and you live in the llama.cpp/Ollama world, AWQ if you serve through vLLM and want robustness, GPTQ if you need the strongest low-bit accuracy and mature kernels. For high-throughput serving on Hopper or Blackwell, FP8 is the rare quant that helps memory and compute at once with almost no accuracy cost, and it should be your starting point there. bitsandbytes NF4 remains the frictionless choice for QLoRA fine-tuning rather than production serving.

Two rules carry most of the weight. Eight bits is nearly free; four bits is the sweet spot; below four bits is a cliff that small models fall off and large models survive — so when VRAM is the constraint, reach for a bigger model at lower precision before a smaller model at higher precision. And the benchmark that matters is your own: perplexity hides real regressions in reasoning, code, and rare languages, so quantize, then evaluate on traffic that looks like yours. Get those two right and quantization is close to a free lunch. Trust a number from someone else’s workload and it is a quiet tax you will not notice until it is in production.

Sources

• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (arXiv 2210.17323)
• AWQ: Activation-aware Weight Quantization for LLM Compression (arXiv 2306.00978)
• SmoothQuant: Accurate and Efficient Post-Training Quantization for LLMs (arXiv 2211.10438)
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (arXiv 2208.07339)
• QLoRA: Efficient Finetuning of Quantized LLMs (arXiv 2305.14314)
• llama.cpp — quantization and GGUF k-quants
• vLLM documentation — supported quantization methods
• NVIDIA — FP8 Formats for Deep Learning (arXiv 2209.05433)
• Hugging Face Transformers — Quantization overview