Fine-tune Llama 3 8B using QLoRA on one 24GB GPU

QLoRA, introduced in the 2023 paper by Dettmers et al., is what actually made fine-tuning multi-billion-parameter models on a single consumer GPU practical. The trick is combining two ideas: quantize the frozen base model to 4-bit so it takes a fraction of the VRAM, and only train small "LoRA" adapter matrices on top of it — so you get gradient updates without ever needing full-precision gradients for all 8 billion parameters.

Setting up a QLoRA fine-tune

1. Load the base model in 4-bit. Hugging Face's transformers library integrates with bitsandbytes to load a model directly in 4-bit NF4 precision — this is what brings an 8B model's footprint down to a size that fits comfortably in 24GB.

2. Attach LoRA adapters via PEFT. Hugging Face's PEFT (Parameter-Efficient Fine-Tuning) library wraps the quantized model with trainable low-rank adapter layers, typically applied to the attention projection matrices.

3. Train on your dataset with a standard trainer loop. Only the adapter weights receive gradient updates — the frozen 4-bit base model is used purely for the forward pass, which is what keeps memory usage manageable.

qlora_finetune.py

# QLoRA setup: 4-bit base model + trainable LoRA adapters
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B",
    quantization_config=bnb_config,
    device_map="auto",
)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],  # attention projections only
    lora_dropout=0.05,
    task_type="CAUSAL_LM",
)

# Wraps the frozen 4-bit model with trainable adapter layers
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()  # typically well under 1% of total params

Approach	Trainable parameters	Approx. VRAM for an 8B model
Full fine-tuning	All ~8 billion	60GB+ (needs data-center GPUs)
QLoRA (4-bit base + LoRA adapters)	Tens of millions (adapters only)	Fits within 24GB

The output of this process is a small adapter, not a new full-size model — to actually deploy it, you either load the base model alongside the adapter at inference time, or merge the adapter weights back into the base model first. Either way, once you have a final merged model, converting it to GGUF (Lesson 3) is the same process as converting any other HuggingFace checkpoint.

Practical Challenge

Look up the PEFT library's print_trainable_parameters() output format and calculate what percentage of an 8B model's parameters a rank-16 LoRA adapter on just the query and value projections actually trains.

Concept Check

What makes QLoRA fine-tuning of an 8B model possible on a single 24GB GPU, when full fine-tuning is not?

Correct! Full fine-tuning needs gradients and optimizer states for every one of the 8 billion parameters. QLoRA freezes and quantizes the base model, so gradient computation only applies to the much smaller LoRA adapter matrices.

Incorrect. Try again! Hint: the memory savings come from which parameters actually need gradients computed, not from changing the model's architecture or where it runs.

Sources & Further Reading

QLoRA: Efficient Finetuning of Quantized LLMs — the original Dettmers et al. paper this technique is named after.
Hugging Face PEFT — Quantization — official docs for combining PEFT's LoRA adapters with quantized base models.
Hugging Face — 4-bit quantization with bitsandbytes and QLoRA — background on the BitsAndBytesConfig setup used in the code above.

Fine-tune Llama 3 8B using QLoRA on one 24GB GPU

Key Insights

Setting up a QLoRA fine-tune

Practical Challenge

Concept Check

Sources & Further Reading