LEARNING HUB
PREDICTIVE TREND INSIGHT
How to track parameter degradation during deep model quantization cycles Illustration

How to track parameter degradation during deep model quantization cycles

Written by Nikhil Badjatya
Direct Summary:

Perplexity comparison, not intuition, is how quantization degradation is actually tracked. llama.cpp ships a dedicated llama-perplexity tool that runs a model over a fixed test set (commonly Wikitext-2) and reports perplexity, plus a --kl-divergence mode that directly compares a quantized model's output distribution against the original FP16 model. llama.cpp's own published benchmarks (LLaMA 3 8B, Wikitext-2) show the actual degradation is small but not zero: Q8_0 adds a delta-PPL of about 0.0027, Q6_K about 0.022, and Q4_K_M about 0.175 — real, measured numbers, not a rounding-error hand-wave.

"The expert in anything was once a beginner."

— Helen Hayes

Key Insights

  • Perplexity delta is the standard degradation metric: compare the quantized model's perplexity on a fixed test set against the FP16 original — a lower delta means less quality lost to quantization.
  • Degradation isn't uniform across quantization levels: llama.cpp's own published numbers on LLaMA 3 8B show roughly a 65x jump in delta-PPL between Q8_0 (~0.0027) and Q4_K_M (~0.175) — "just use the smallest one that fits" isn't free.
  • KL-divergence gives a second, complementary signal: llama.cpp's --kl-divergence mode compares token-level output distributions directly, catching shifts that an aggregate perplexity number can smooth over.

"How much quality did I lose?" is an empirical question, and llama.cpp already ships the tool to answer it precisely rather than by feel. The workflow is: run perplexity on the original FP16 model, run it again on each quantized candidate, and compare.

Measuring degradation with llama.cpp's perplexity tool

1. Establish your FP16 baseline. Run llama-perplexity against the unquantized model on a fixed test corpus (Wikitext-2 is the common default) and record the baseline perplexity score.

2. Run the same test against each quantized candidate. Quantize to your candidate formats (Q8_0, Q6_K, Q4_K_M, etc.) and run the identical perplexity test on each — same corpus, same context length, so the comparison is apples-to-apples.

3. Compare delta-PPL, and cross-check with KL-divergence for borderline cases. The gap between baseline and quantized perplexity is your degradation number; llama.cpp's --kl-divergence-base/--kl-divergence flags add a token-distribution-level comparison if you need more resolution than an aggregate score gives you.

measure_quant_degradation.sh 
# 1. Baseline perplexity on the unquantized FP16 model
./llama-perplexity \
  --model ./models/model-fp16.gguf \
  --file ./wikitext-2-raw/wiki.test.raw

# 2. Same test on a quantized candidate
./llama-perplexity \
  --model ./models/model-q4_k_m.gguf \
  --file ./wikitext-2-raw/wiki.test.raw

# 3. Optional: token-distribution comparison vs. the FP16 baseline
./llama-perplexity \
  --model ./models/model-q4_k_m.gguf \
  --kl-divergence-base ./fp16-logits.kld \
  --kl-divergence
Quantization Level (LLaMA 3 8B, Wikitext-2) Delta-PPL vs. FP16 (llama.cpp benchmark)
Q8_0 ~0.002650
Q6_K ~0.021748
Q4_K_M ~0.175482 (± 0.004620)

These are llama.cpp's own repo-committed benchmark figures, not third-party estimates — and the project's own documentation cautions they're implementation-specific, so don't treat them as universal constants across every model family or quantization tool. Re-running the perplexity test on your specific model and target quantization level is what actually tells you whether the degradation is acceptable for your use case.

Practical Challenge

Run llama-perplexity against an FP16 model and a Q4_K_M quantization of the same model on the same test file, and calculate the delta-PPL yourself.

Concept Check

Based on llama.cpp's own published LLaMA 3 8B benchmarks, how does Q4_K_M's degradation compare to Q8_0's?
Correct! llama.cpp's published numbers show a substantial gap — Q4_K_M trades meaningfully more quality for its smaller size than Q8_0 does.
Incorrect. Try again! The published benchmark shows a clear, non-trivial gap between the two — Q8_0 preserves quality far better than Q4_K_M, at the cost of more VRAM.

Sources & Further Reading

  • llama.cpp: Perplexity tool README — documents the llama-perplexity tool, its --kl-divergence mode, and the source of the Q8_0/Q6_K/Q4_K_M delta-PPL benchmark figures cited above.
Previous Guide Dashboard Next Guide