Semantic caching for AI applications: what to cache, and what it costs you when wrong

Caching in front of a language model is unusual: it is the only optimisation that improves latency, cost and throughput simultaneously. It is also the only one that can return a confidently wrong answer, because a semantic cache decides that two different questions are the same question.

That trade-off is the whole subject.

Four layers, cheapest first

Exact match earns its place before anything clever

Normalise — lowercase, collapse whitespace, strip trailing punctuation — and hash. In support and internal-documentation workloads the repetition rate is far higher than teams expect, because people ask the same questions in the same words.

const key = createHash('sha256')
  .update(`${model}:${systemPromptVersion}:${normalise(question)}`)
  .digest('hex');

Embedding cache is free money

Re-embedding unchanged documents on every ingest is pure waste and is the default behaviour of most naive pipelines. Key on a hash of the chunk text plus the model name.

The semantic layer, and the threshold that decides everything

A semantic cache embeds the incoming question and returns a previous answer if a stored question is close enough. "Close enough" is one number, and it is the most consequential parameter in the system.

const embedding = await embed(question);
const [nearest] = await vectorStore.search(embedding, { limit: 1 });
 
if (nearest && nearest.score >= THRESHOLD) {
  metrics.increment('cache.semantic.hit', { score: nearest.score });
  return nearest.answer;
}

Three practices that make it safe rather than exciting:

1. Start conservative and log near-misses

   Begin around 0.95 cosine similarity. Log every query scoring between 0.85 and your threshold along with what it would have returned, and read a sample by hand. That set is what tells you whether you can safely lower it.

2. Scope the cache key by everything that changes the answer

   Tenant, user role, locale, permission set. A cached answer crossing a tenant boundary is a data-leak incident, not a cache miss.

3. Never semantically cache anything with a personal or live value

   "What is my balance?" and "What is my current balance?" are semantically identical and must never share an answer. Route personalised questions past the cache entirely.

Invalidation is the part nobody plans

A cache in front of a RAG system holds answers derived from documents. When a document changes, which cached answers are now wrong?

1. Record the source chunks with every cached answer

   Then a document update invalidates every answer that cited it. This is the only precise approach, and it requires you to have been storing citations — which you should be doing for correctness anyway.

2. Give every entry a TTL as a backstop

   Short for anything volatile, longer for stable reference material. TTL alone is not invalidation; it is a bound on how long you can be wrong.

3. Version the whole cache with the prompt and the model

   A prompt change should invalidate everything. Including the version in the key makes that automatic rather than a deployment step somebody forgets.

Where to put it

Redis or Valkey with vector search covers all four layers in one system, which matters because a cache that needs its own infrastructure often does not get built.

// Exact and semantic in one store; the exact lookup runs first because it is free.
const exact = await redis.get(`ans:${exactKey}`);
if (exact) return JSON.parse(exact);
 
const near = await redis.ft.search('idx:answers', `*=>[KNN 1 @vec $q AS score]`, {
  PARAMS: { q: Buffer.from(new Float32Array(embedding).buffer) },
  DIALECT: 2,
});

Keep the answer, the source citations, the model, the prompt version and the creation time in the entry. Everything except the answer exists so you can invalidate correctly.

Measure the right things

The short version

1. Exhaust exact, prefix and embedding caching before going semantic

   Three layers with no correctness risk, and more traffic than expected.

2. Start the semantic threshold high and lower it on evidence

   Log near-misses; read them; then decide.

3. Scope every key by tenant, role and locale

   A cross-tenant hit is an incident.

4. Never semantically cache personalised or live answers

   Route them past the cache.

5. Store citations so invalidation can be precise, and version by prompt and model

   TTL is a bound on being wrong, not a substitute for invalidation.