Quantization: shrinking models
How a model gets 3–4× smaller with only a small loss in quality.
3 min · you’ll learn
- What Q4, Q8 and FP16 mean
- Which quant to download
- Bigger-at-Q4 vs. smaller-at-Q8
Storing each number with fewer bits
Quantization shrinks a model by storing each of its billions of numbers less precisely. Instead of 16 bits per number, you might use 8 or 4.
Think of it like this
It’s like saving a photo as a JPEG. The file gets much smaller, and unless you zoom right in you barely notice the difference. Squeeze too hard, though, and it starts to look blocky.
The labels you’ll see
When you download a model you’ll pick a version like Q4_K_M or “4-bit”. The number is roughly the bits per parameter:
Size of the same 9B model
- FP16 / BF16full precision≈18 GB
- Q8practically no loss≈9.5 GB
- Q4the sweet spot≈5.7 GB
- Q3starts to hurt≈4.7 GB
- Q8: half the size of full precision, quality basically identical.
- Q4: the usual choice. About 30% of full size with a small quality loss.
- Q3 and below: saves more memory but quality drops, especially for coding and small models.
MXFP4 is a 4-bit format some models (like gpt-oss) are released in directly, so it carries no extra loss.
Try it: how big will a model be?
Live · uses the real engine
Quantization size calculator
Weights only (GGUF). KV cache and runtime overhead come on top.
Q4 (Q4_K_M) ≈ 16 GBSlight loss
The usual sweet spot: roughly a quarter of FP16 size with small quality loss.
Click a row for details. Sizes use published file sizes where known, otherwise parameters × bits per weight. Quality notes are editorial approximations.
Smaller is also faster
Remember: the computer re-reads the model for every word. A Q4 file is a quarter the size of FP16, so there are far fewer bytes to read and words come out faster.
That’s why a bigger model at Q4 usually beats a smaller model at Q8: similar memory, but more knowledge.
Remember this
Quick check
You have room for about 18 GB. Which is usually the better choice?
Up next · lesson 5
Why “it fits” isn’t enough
A model can fit in memory and still make your computer miserable to use.