How to run local embeddings models on Raspberry Pi for smart home automation

It's worth being direct about what actually powers most smart-home voice control before reaching for embeddings at all. Home Assistant's Assist pipeline, when set to prefer local intents, resolves phrases like "turn off the kitchen lights" through hassil, a template-matching engine — not a neural network. Sentences are defined as YAML templates (the Home Assistant Intent Language, or HassIL) and matched directly. It's fast, deterministic, and needs no embedding model for the common case.

Where local embeddings actually help

The gap shows up when a spoken command doesn't fit any defined template — a user says "kill the lights in the kitchen" instead of "turn off," or refers to a device by a nickname that was never added to the template list. Rather than failing outright or falling back to a cloud LLM, a small local embeddings model can compare the unmatched phrase against your known device and scene names, and route to whichever is semantically closest — entirely on-device, with no cloud round-trip and no added command latency the user would notice.

Running it on a Pi 5's CPU

1. Pick a small model. A compact sentence-transformers model like all-MiniLM-L6-v2 is a reasonable starting point — it's small enough to run comfortably on CPU.

2. Install the ONNX backend: pip install sentence-transformers[onnx]. Sentence Transformers can load or auto-convert a model to ONNX format, and ONNX shows measurable CPU speedups over the default PyTorch backend, especially on short text like device names.

3. Quantize for ARM. The export_dynamic_quantized_onnx_model() function supports an arm64 quantization config specifically for ARM CPUs like the Pi 5's Cortex-A76 — dynamic int8 quantization needs no calibration dataset and further cuts inference time.

# Fallback fuzzy-match for phrases hassil's templates didn't catch,
# running on ONNX so it's fast enough on a Pi 5's CPU
from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim

model = SentenceTransformer(
    "sentence-transformers/all-MiniLM-L6-v2",
    backend="onnx"
)

known_devices = ["kitchen lights", "living room lamp", "hallway light"]
device_embeddings = model.encode(known_devices)

# Unmatched phrase from hassil's template engine
query = "kill the lights in the kitchen"
query_embedding = model.encode(query)

scores = cos_sim(query_embedding, device_embeddings)[0]
best_match = known_devices[scores.argmax()]
print(f"Routing to: {best_match} (score: {scores.max():.2f})")

The honest framing is that embeddings are a narrow, useful patch for the edges of your voice pipeline — not a replacement for the deterministic matching that already handles most traffic reliably. Reach for a local embedding model when you've actually observed unmatched commands failing, not as a default architecture choice.