LEARNING HUB
PREDICTIVE TREND INSIGHT
Fine-tuning local reasoning models using synthetic data generation Illustration

Fine-tuning local reasoning models using synthetic data generation

Written by Nikhil Badjatya
Direct Summary:

The proven way to give a small local model reasoning ability is distillation via synthetic reasoning traces: use a strong reasoning model to generate chain-of-thought solutions for your target task, then supervised-fine-tune (SFT) a smaller model on those traces — no reinforcement learning required. DeepSeek-R1's own team validated this at scale: they SFT'd Qwen and Llama models from 1.5B to 70B parameters on 800k reasoning traces generated by R1 itself, and found this distillation approach outperformed applying RL directly to the small model.

"The science of today is the technology of tomorrow."

— Edward Teller

Key Insights

  • Distillation beats direct RL for small models: DeepSeek-R1's paper reports that SFT-ing a small model on a large reasoning model's traces produced better results than running RL training on the small model directly.
  • You need a teacher and a student: a large, already-capable reasoning model generates the synthetic chain-of-thought traces; a smaller model (the one you'll actually run locally) is fine-tuned on them.
  • Unsloth makes the student-side training tractable on consumer hardware: its QLoRA implementation is built specifically to cut VRAM and training time for exactly this kind of local fine-tuning run.

Training reasoning ability into a small model from scratch with reinforcement learning is expensive and finicky. The distillation shortcut — let a bigger model do the hard reasoning work once, capture its chain-of-thought as training data, then SFT a small model to imitate that pattern — is both cheaper and, per DeepSeek's own published results, more effective for small models specifically.

Generating and training on synthetic reasoning traces

1. Generate traces with a strong reasoning model. Prompt a capable reasoning model (locally via Ollama, or through an API) to solve problems representative of your target task step-by-step, and save the full chain-of-thought plus final answer as your training examples.

2. Format the traces as SFT training data. Each example becomes a prompt/completion pair where the completion is the full reasoning trace, not just the final answer — the model needs to learn the reasoning process, not memorize outputs.

3. Fine-tune the student model with Unsloth's QLoRA pipeline. Unsloth wraps 4-bit QLoRA training with kernel optimizations that reduce both VRAM use and wall-clock time versus a standard Hugging Face PEFT training loop, making this practical on a single consumer GPU.

distill_reasoning.py 
# SFT a small student model on synthetic reasoning traces, via Unsloth
from unsloth import FastLanguageModel
from trl import SFTTrainer

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Qwen2.5-7B-bnb-4bit",
    max_seq_length=4096,
    load_in_4bit=True,
)

# Add LoRA adapters -- only these small matrices get trained
model = FastLanguageModel.get_peft_model(
    model, r=16, lora_alpha=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
)

# reasoning_traces_dataset: prompt + full CoT trace as completion
trainer = SFTTrainer(model=model, tokenizer=tokenizer, train_dataset=reasoning_traces_dataset)
trainer.train()
Approach for Small-Model Reasoning Result (per DeepSeek-R1 report)
Apply RL directly to the small model Weaker — small models struggle to discover reasoning patterns via RL alone
SFT the small model on a larger model's synthetic reasoning traces Stronger — the small model imitates already-discovered reasoning patterns

The practical takeaway isn't just "distillation works" — it's that for a small model specifically, imitating a larger model's reasoning traces is the higher-leverage move compared to trying to train reasoning ability into it directly. That's exactly the workflow Unsloth's QLoRA tooling is built to make affordable on local hardware.

Practical Challenge

Generate 20 chain-of-thought traces from a reasoning model for a task you care about (e.g., math word problems), format them as prompt/completion pairs, and run a short Unsloth QLoRA fine-tune on a small student model.

Concept Check

Per DeepSeek-R1's published results, how did SFT-distillation on synthetic reasoning traces compare to applying RL directly to a small model?
Correct! DeepSeek-R1's report found distillation from a larger reasoning model's traces gave better results for small models than training them with RL directly.
Incorrect. Try again! The published result favors distillation — small models learn reasoning better by imitating a larger model's traces than by discovering it themselves via RL.

Sources & Further Reading

Previous Guide Dashboard Next Guide