AI
"Running DeepSeek R1 locally" almost always means running one of the distilled variants — smaller models fine-tuned on R1's reasoning traces, not the full 671B-parameter model, which needs enterprise-grade multi-GPU hardware. The distilled 7B/8B/14B versions are realistic on a single consumer GPU once quantized, and Ollama is the easiest on-ramp for that.
Step-by-Step Implementation
1. Retrieve Model Weights: Download models from HuggingFace and verify hash checksums.
2. Execute Quantization Script: Compile weights into GGUF layout using conversion tools to fit model parameters within VRAM boundaries.
3. Launch Local Endpoint: Start the local runner on your workstation and configure tool bindings.
# Shell command guide to run local SLMs on consumer hardware
# 1. Pull DeepSeek-R1 distilled 8B model via Ollama
ollama run deepseek-r1:8b
# 2. Host custom local GGUF model via llama-server
# Set context size to 8k and map layers directly to GPU VRAM
./llama-server \
--model ./models/deepseek-distill-q4_k_m.gguf \
--ctx-size 8192 \
--n-gpu-layers 33 \
--port 8080| Quantization Level | VRAM Requirement | Reasoning Loss |
|---|---|---|
| FP16 (Uncompressed) | High (~16GB for an 8B model) | 0% (Baseline performance) |
| Q4_K_M (4-bit Medium) | Low (~6GB for an 8B model) | Negligible (Recommended for standard GPUs) |
Ollama handles the quantization and serving details for you (it pulls a pre-quantized GGUF under the hood); the llama-server route in the second command gives more control if you've converted or quantized a model yourself, which is what Lesson 3 covers.
Practical Challenge
Deploy Ollama, pull a 3B or 8B model, and run a script to output token generation speeds (tokens per second).
Concept Check
Sources & Further Reading
- Ollama Documentation — official docs for the
ollama runcommand and model library used above. - llama.cpp (GitHub) — the inference engine behind
llama-serverand the GGUF format itself.