Building a local caching layer to prevent duplicate API token costs

Before reaching for anything sophisticated, it's worth asking a simpler question: how many of your API calls are literally identical requests you've already paid for once? Retried requests after a transient error, duplicate submissions from a UI bug, or repeatedly re-running the same analysis during development are all genuinely common, and all of them can be caught with a straightforward hash-based cache — no embeddings, no similarity thresholds, just an exact key lookup.

Building the exact-match cache

1. Hash the full request, not just the user's message. Include the model name and any system prompt in the hash — the same user question against a different model or system prompt is a genuinely different request and shouldn't share a cache entry.

2. Check the cache before calling the API, write to it after. A cache miss means: call the API, store the response keyed by the request hash, then return it — subsequent identical requests hit the cache instead.

3. Set a reasonable expiration. Unlike a RAG knowledge base, API response caches often benefit from a time-to-live (hours to days, depending on your use case) so stale cached answers don't linger indefinitely if the underlying data changes.

# pip install diskcache
import hashlib, json
from diskcache import Cache

cache = Cache("./api_cache")

def request_hash(model: str, messages: list) -> str:
    payload = json.dumps({"model": model, "messages": messages}, sort_keys=True)
    return hashlib.sha256(payload.encode()).hexdigest()

def cached_completion(model: str, messages: list, call_api_fn):
    key = request_hash(model, messages)
    if key in cache:
        return cache[key]  # cache hit -- zero additional API cost
    result = call_api_fn(model, messages)
    cache.set(key, result, expire=86400)  # 24-hour TTL
    return result

Start with exact-match caching — it's a few lines of code and catches a genuinely common cost leak (retries, duplicate UI submissions, repeated dev runs) with zero risk of returning a wrong-but-similar cached answer. Layer semantic caching on top only once you've confirmed paraphrased duplicates are a meaningful share of your traffic.