LEARNING HUB
PREDICTIVE TREND INSIGHT
Comparing embedding latency profiles across local NVMe storage clusters Illustration

Comparing embedding latency profiles across local NVMe storage clusters

Written by Nikhil Badjatya
Direct Summary:

To optimize comparing embedding latency profiles across local NVMe storage clusters, developers build localized index layers (using HNSW or IVF architectures) inside databases like pgvector or ChromaDB. These systems map text embeddings into multi-dimensional coordinates, allowing sub-millisecond similarity lookups.

"Any sufficiently advanced technology is indistinguishable from magic."

— Arthur C. Clarke

Key Insights

  • Index Parameters: HNSW indexes offer fast retrieval speeds at scale, but require more memory and setup time compared to flat vector scans.
  • VRAM Caching: Store common query vectors in a local memory cache to bypass model calculation latency for repeated searches.
  • Dimensionality Matching: Ensure search query models output the exact coordinate count (e.g. 384 or 1536 dimensions) matching the database.

HNSW and IVFFlat, covered earlier in this course, both assume the index fits in RAM. That assumption breaks at billion-scale: a billion 768-dimensional float32 embeddings alone is roughly 3TB, before the graph or cluster structure adds anything on top. Buying enough RAM to hold that is possible but expensive. The alternative that's actually shipped in production is DiskANN, Microsoft's disk-based approximate nearest neighbor system, which deliberately keeps most of the index on NVMe SSD instead of in memory.

The Actual Mechanism

DiskANN's original research paper describes indexing, storing, and searching a billion-point database "on a single workstation with just 64GB RAM and an inexpensive solid-state drive." The trick isn't that SSDs are as fast as RAM — they're not — it's a split: a compressed version of every vector stays in RAM for a fast first-pass filter, while the graph structure and full-precision vectors live on the NVMe drive, read only for the small number of candidates the compressed search narrows things down to. On the standard billion-point SIFT1B benchmark, the paper reports over 5,000 queries per second at under 3ms mean latency and 95%+ recall on a 16-core machine — numbers that depend heavily on NVMe's random-read performance being fast enough that those disk reads don't dominate the query.

Why This Only Works Well on NVMe, Not Spinning Disk or SATA SSD

The "sub-millisecond disk read" part of this equation is the whole reason the technique is viable. A query in DiskANN's graph traversal issues several small, effectively-random reads per hop as it walks the graph on disk. NVMe drives, which connect over PCIe and support deep queued parallel I/O, handle that access pattern at latencies low enough (tens of microseconds) that it doesn't dominate total query time. The same access pattern against a spinning hard drive, with its physical seek time, would be unusably slow — this is specifically an NVMe-era technique, not a generic "use disk instead of RAM" trick.

diskann_tradeoff.txt 
# Rough shape of the trade-off DiskANN is built around:
#
# All-in-RAM HNSW:     fastest per-query, but RAM cost scales
#                       linearly with dataset size
# DiskANN (RAM+NVMe):  compressed vectors in RAM for filtering,
#                       full vectors + graph read from NVMe on demand
#                       -> ~10x+ more vectors served per machine,
#                       at a real but small added per-query latency
Storage Tier Typical Random-Read Latency Fit for Graph-Traversal ANN
RAM ~100 nanoseconds Fastest, but capacity-limited and expensive at billion-vector scale
NVMe SSD Tens of microseconds What DiskANN is built for — fast enough for per-hop graph reads
SATA SSD / spinning disk Hundreds of microseconds to several milliseconds Too slow for the random per-hop access pattern graph traversal needs

The practical takeaway for anyone sizing a local vector search cluster: if your dataset comfortably fits in RAM, a standard in-memory HNSW index (as covered earlier in this course) is simpler and faster per query. DiskANN and disk-based ANN techniques earn their added complexity specifically once RAM cost becomes the binding constraint — and even then, they depend on NVMe-class storage to keep the added disk-read latency small enough to matter.

Practical Challenge

Estimate the RAM cost of storing 100 million 768-dimensional float32 embeddings in memory (dimensions × 4 bytes × count), then compare that to what an NVMe drive of the same capacity costs — this is the actual economic trade-off DiskANN-style systems are built to address.

Concept Check

Why does DiskANN specifically require NVMe-class storage rather than any SSD or hard drive?
Correct! The graph-traversal access pattern is many small random reads per query. NVMe's low random-read latency (tens of microseconds) is what makes reading the graph from disk viable instead of a bottleneck.
Incorrect. Try again! Hint: think about the specific read pattern a graph traversal produces — many small, scattered reads — and which storage tier handles that pattern fastest.

Sources & Further Reading

Previous Guide Dashboard Next Guide