LEARNING HUB
CURRENT TREND INSIGHT
How to convert huggingface model weights to GGUF format Illustration

How to convert huggingface model weights to GGUF format

Written by Nikhil Badjatya
Direct Summary:

Converting a HuggingFace model to GGUF means running llama.cpp's convert_hf_to_gguf.py script against the model's safetensors/config files to produce an FP16 GGUF, then optionally running llama-quantize on that file to shrink it to 4-bit or 8-bit. The two steps are separate on purpose — you always convert to a full-precision GGUF first, then quantize down from there, rather than quantizing directly from the HuggingFace format.

"Change is the end result of all true learning."

— Leo Buscaglia

Key Insights

  • Convert first, quantize second: convert_hf_to_gguf.py produces an FP16 (or BF16) GGUF from the original weights. Quantizing is a separate, later step run against that file with llama-quantize.
  • Not every architecture is supported: the conversion script maintains an explicit list of supported model architectures — check llama.cpp's repo before assuming a brand-new HuggingFace model will convert cleanly.
  • The tokenizer has to convert too: a mismatched or incorrectly converted tokenizer is one of the most common sources of garbled output after conversion — verify the vocab size in the GGUF metadata matches the source model.

Lesson 2 covered running an already-quantized model through Ollama or llama-server. This lesson covers where that GGUF file actually comes from when you're starting with a model straight from HuggingFace — the two-step convert-then-quantize process that llama.cpp itself uses.

The actual conversion pipeline

1. Clone llama.cpp and install its Python requirements. The conversion script is part of the llama.cpp repository, not a standalone package — you need the repo checked out to run it.

2. Run convert_hf_to_gguf.py against the HuggingFace model directory. This reads the model's config, tokenizer, and safetensors weights and writes out a single FP16 (or BF16) GGUF file — no quantization yet, just a format change.

3. Quantize the FP16 GGUF down with llama-quantize. This is the separate step that actually shrinks the file — Q4_K_M for a good size/quality balance, Q8_0 if you want to stay closer to full precision.

convert_and_quantize.sh 
# 1. Convert HuggingFace weights to an FP16 GGUF (llama.cpp repo required)
python llama.cpp/convert_hf_to_gguf.py \
  ./my-model-hf-directory \
  --outtype f16 \
  --outfile ./my-model-f16.gguf

# 2. Quantize the FP16 GGUF down to 4-bit (separate step, separate tool)
./llama-quantize \
  ./my-model-f16.gguf \
  ./my-model-q4_k_m.gguf \
  Q4_K_M
Step Tool What changes
1. Format conversion convert_hf_to_gguf.py HuggingFace safetensors → single GGUF file, still full precision
2. Quantization llama-quantize Full-precision GGUF → smaller quantized GGUF (e.g. Q4_K_M)

Once you have the quantized GGUF, it's the exact file format Lesson 2's Ollama and llama-server commands expect — this is the step that produces the .gguf file those tools load.

Practical Challenge

Pick a small HuggingFace model (under 2B parameters) confirmed on llama.cpp's supported-architectures list, convert it to FP16 GGUF, then quantize it to both Q4_K_M and Q8_0 and compare the resulting file sizes.

Concept Check

Why does converting a HuggingFace model to a quantized GGUF require two separate steps instead of one?
Correct! The conversion script's job is translating HuggingFace's format into GGUF at full precision; quantization is a distinct, later operation performed by a different tool on that GGUF file.
Incorrect. Try again! Hint: think about what each tool's name actually says it does — "convert" vs. "quantize" are two different jobs.

Sources & Further Reading

Previous Guide Dashboard Next Guide