Building vector cache layers to reduce token costs in production RAG

The naive way to "cache" LLM responses is a hash map keyed on the exact query string — which almost never hits, because real users rarely phrase the same question identically twice. A semantic cache fixes this by embedding the incoming query and checking a vector index for anything close enough to a previously-answered question; on a hit, you return the cached response and skip the model call (and its token cost) entirely.

Step-by-Step Implementation

1. Embed the incoming query. Use the same embedding model you'd use for RAG retrieval — the cache and your retrieval pipeline can often share infrastructure.

2. Check the vector index for a near match. An HNSW-indexed store returns the closest cached query and its similarity score in sub-millisecond time, even at large cache sizes.

3. Serve the cached response above a similarity threshold, otherwise call the LLM. Tune the threshold carefully — too loose and you serve wrong answers to different questions; too strict and you rarely get a cache hit at all.

# Semantic cache lookup: check for a similar prior query before calling the LLM
import sqlite3
import numpy as np

def cosine_similarity(v1, v2):
    return np.dot(v1, v2) / (np.linalg.norm(v1) * np.linalg.norm(v2))

SIMILARITY_THRESHOLD = 0.92  # tune based on false-hit tolerance

def get_cached_or_call_llm(query, embed_fn, cache_rows, call_llm_fn):
    query_vec = embed_fn(query)

    best_score, best_response = 0.0, None
    for cached_query, cached_vec, cached_response in cache_rows:
        score = cosine_similarity(query_vec, cached_vec)
        if score > best_score:
            best_score, best_response = score, cached_response

    if best_score >= SIMILARITY_THRESHOLD:
        return best_response  # cache hit - zero LLM tokens spent

    # Cache miss - call the model and store the result for next time
    response = call_llm_fn(query)
    return response

The real production version of this (GPTCache is a ready-made example) also needs a real vector index rather than the linear scan shown above — at cache sizes past a few thousand entries, an HNSW-backed store keeps lookup fast enough that the cache check itself never becomes the bottleneck.