LEARNING HUB
PREDICTIVE TREND INSIGHT
How to configure real-time voice-to-voice models on single consumer GPUs Illustration

How to configure real-time voice-to-voice models on single consumer GPUs

Written by Nikhil Badjatya
Direct Summary:

Kyutai's Moshi is a genuinely open-source, single-model speech-to-speech system — it listens and talks back through one 7-billion-parameter model instead of chaining separate transcription, chat, and text-to-speech steps. The unquantized PyTorch version needs a GPU with roughly 24GB of VRAM, but Kyutai also ships quantized (int8/bf16) builds on a Rust/Candle backend that run on smaller consumer GPUs and Apple Silicon via Metal. Kyutai reports a theoretical 160ms round-trip latency and roughly 200ms practical latency on a data-center L4 GPU — real numbers from their own repo, not a marketing estimate.

"If you can't explain it simply, you don't understand it well enough."

— Albert Einstein

Key Insights

  • Moshi is one model, not a pipeline: unlike a setup that chains Whisper (speech-to-text) → an LLM → a separate TTS model, Moshi processes audio in and audio out through a single 7B-parameter model, which is what makes its low round-trip latency possible.
  • 24GB VRAM for the reference PyTorch build: Kyutai's own README states quantization isn't supported on the PyTorch backend, so that path needs a GPU with "a significant amount of memory (24GB)" — think an RTX 3090/4090 class card, not an 8GB laptop GPU.
  • The Rust/Candle backend is the consumer-hardware path: it ships q8 and bf16 quantized weights and supports Apple Silicon via Metal, which is the realistic route for running Moshi on a single consumer GPU or a MacBook rather than a 24GB workstation card.

A voice-to-voice model that "just talks back" sounds simple, but most real-time voice assistants are actually three separate models stitched together: speech-to-text, a language model, and text-to-speech, each adding its own processing delay. Kyutai's Moshi, released as open source by the French AI lab Kyutai, takes a different approach — one unified model that processes raw audio input and produces raw audio output directly, which is what lets it hold a fluid back-and-forth conversation instead of a stop-and-wait exchange.

What it actually takes to run Moshi locally

1. Pick your backend based on your GPU. Kyutai's own repository is explicit that the PyTorch implementation does not support quantization, so it needs a GPU with roughly 24GB of VRAM — a single high-end consumer card (RTX 3090/4090 class), not a data-center cluster, but also not a modest laptop GPU.

2. Use the Rust/Candle backend for smaller hardware. Kyutai also publishes q8 and bf16 quantized weights on a Rust/Candle backend, which is the practical path if you don't have 24GB of VRAM free. This backend also supports Apple Silicon: swapping the `--features cuda` build flag for `--features metal` runs Moshi on a Mac's integrated GPU instead.

3. Expect real-time latency, not instant response. Kyutai states a theoretical latency of 160ms (an 80ms audio frame plus 80ms of acoustic delay from its Mimi codec), with practical end-to-end latency around 200ms measured on an Nvidia L4 GPU — fast enough to feel conversational, but not literally zero.

moshi_setup.sh 
# Rust/Candle backend — the practical route for a single consumer GPU
# (quantized weights; PyTorch backend needs ~24GB VRAM unquantized)

git clone https://github.com/kyutai-labs/moshi
cd moshi/rust

# Linux/Windows with an Nvidia GPU (requires nvcc / CUDA toolkit):
cargo build --features cuda --release

# macOS with Apple Silicon, using Metal instead of CUDA:
cargo build --features metal --release

# Weights (Moshika/Moshiko, q8 or bf16) are pulled from Kyutai's
# Hugging Face repos referenced in the project README.
Backend Precision Realistic hardware
PyTorch No quantization support ~24GB VRAM GPU (e.g. RTX 3090/4090)
Rust/Candle (CUDA) q8 / bf16 quantized Smaller consumer Nvidia GPUs
Rust/Candle (Metal) q8 / bf16 quantized Apple Silicon Mac (integrated GPU)

The practical takeaway is that "runs on a single consumer GPU" depends heavily on which backend you pick — the reference PyTorch implementation is the least forgiving option, while the quantized Rust backend is what actually makes this a MacBook- or single-consumer-GPU-friendly project, matching how Kyutai itself frames the smaller variant.

Practical Challenge

Check your own GPU's VRAM (Task Manager on Windows, `nvidia-smi` on Linux, or System Report on a Mac), then decide from the table above which Moshi backend you'd actually be able to run — and whether the Rust/Metal path is the realistic option for your hardware.

Concept Check

Why can Moshi hold a fluid, low-latency conversation compared to a chained STT-then-LLM-then-TTS voice assistant?
Correct! Moshi's unified audio-in/audio-out architecture avoids the extra latency of converting audio to text, running a separate LLM, then converting text back to audio.
Incorrect. Try again! Moshi can run locally on consumer hardware (via the quantized Rust backend), and it does generate genuine responses rather than replaying pre-recorded audio — its speed comes from being a single unified model.

Sources & Further Reading

  • kyutai-labs/moshi (GitHub) — official repository; source for the 24GB PyTorch VRAM requirement, the Rust/Candle q8/bf16 quantized backend, CUDA vs. Metal build flags, and the 160ms/200ms latency figures.
  • Kyutai — Speech AI — Kyutai's own project pages describing Moshi and its speech-to-speech research direction.
Previous Guide Dashboard Next Guide