Caching an answer nobody asked for exactly

How do you reuse a stored answer for a question that has never been asked in those words?

An ordinary cache is a dictionary. You hash the key, you find the entry, you serve it. That works beautifully when keys repeat — the same URL, the same query text, the same file path — because the world hands you the identical key again and again.

Now key a cache on what a person typed. "How do I reset my password?" "how to reset password" "I forgot my password, what now". Three keys, one intent, zero hits. The exact-match cache is not slightly less effective here; it is close to useless. So what would it take to hit anyway — and what have you quietly given up the moment you try?

The first move is to stop keying on the characters and start keying on something that stays put when the wording moves. An embedding model does exactly that: it maps a piece of text to a vector, trained so that texts people judge similar land near each other. Store each answered question as a vector with its answer beside it; when a new question arrives, embed it too and search for the nearest stored vector. If the nearest one is close enough, serve its answer without calling the expensive model at all. That is the whole mechanism — and open implementations such as GPTCache do essentially this.

But notice the phrase doing all the work: close enough. There is no natural boundary in that space, so you must choose a similarity threshold, and the choice is not a tuning detail. It is the entire design, because it is a dial between two different errors. Set it high and you hit rarely, and the cache barely earns the vector search it costs. Set it low and you begin answering questions nobody asked.

Sit with that second error, because it is not the familiar cache problem. A conventional cache's failure is stale — the right entry, out of date. This one's failure is wrong entry, confidently served, and it looks perfectly plausible to the reader. Worse, the errors are not randomly distributed. Ask yourself which pairs of sentences are close in an embedding space but opposite in meaning. "How do I enable two-factor authentication" and "how do I disable two-factor authentication" differ by one morpheme. So do questions differing only in a number, a date, a name, or a negation — and these are exactly the edits that flip the correct answer while barely moving the vector. Embedding models trained on similarity objectives are known to handle negation poorly; how badly varies by model, and it is an active area of work rather than a solved one, but the direction of the weakness is well established.

There is a second problem hiding in the key. Is the question really the whole key? "What is my balance" means something different for each user. So does any question whose answer depends on the tenant, the permissions of the asker, the retrieved documents, the system prompt, the tool set, or the model version. If those are not part of the key, the cache does not merely go stale — it serves one user's answer to another, which is a data-leak rather than a performance bug. The honest key is the embedding plus every scope the answer depended on, and scopes that are hashed exactly rather than compared by similarity.

Then the ordinary caching problems arrive on top: entries age, so a stored answer about a price, a policy, or a schedule must expire; and a hit is not free, since it costs an embedding call and an approximate-nearest-neighbour search. Semantic caching pays when the call being avoided is far more expensive than that — which for a large model it usually is, in both money and latency.

Is there a way out of the threshold dilemma? Partly. The strongest practical move is to stop treating similarity as the decision and treat it as a shortlist: retrieve the few nearest candidates, then have something cheap actually check equivalence — a small model asked "do these two questions have the same answer?", or a rule that refuses any pair differing in a negation, a numeral, or a named entity. That converts one fuzzy score into a retrieval step plus a verification step, which is a much better-behaved shape.

Three refinements.

First, the threshold is a classifier boundary, not a preference. Every value of it produces a rate of missed reuse and a rate of wrong reuse, and you cannot lower one without raising the other from the same evidence. That means it should be chosen against a labelled set of question pairs you have actually judged, and re-checked whenever the embedding model changes — similarity scores are not comparable across models.

Second, the two errors are not equally costly. A missed hit costs money; a wrong hit costs trust, and in some domains a great deal more. That asymmetry argues for a high threshold plus a verification step, and for excluding whole classes of question — anything personal, live, or high-stakes — rather than hunting for one number safe for all of them.

Third, a common misconception: that a hit means the questions were the same. It means their representations were near. Those are different claims, and the gap between them is where every interesting failure of this technique lives.

• Approximate nearest-neighbour indexes, and how their own recall setting compounds with the threshold.
• Verification with a cheap model, and whether it can be trusted to judge equivalence.
• Cache scoping in multi-tenant systems, where a wrong hit becomes a disclosure rather than an error.

Every term the collection defines is gathered in the glossary.