Exact caching and semantic caching
An exact cache hashes the request and returns the stored response only when the hash matches. It is safe: identical bytes, identical answer. It also misses every rephrase. explain mutexes and please explain mutexes are different keys.
A semantic cache tries to catch those rephrases. The common design embeds the prompt, searches a vector store for the nearest stored prompt, and calls it a hit when the similarity clears a threshold.
| Tool | How a hit is decided | Threshold (documented) |
|---|---|---|
| GPTCache | Embedding search plus a pluggable evaluator | similarity_threshold default 0.8 |
| Portkey semantic cache | Cosine similarity after an exact miss | Default 0.95 |
| Bifrost | Exact hash, then cosine similarity | Default 0.8 |
| LiteLLM redis-semantic | Cosine similarity | Docs examples use 0.8 |
| MorrowCache | Exact hash, then a shortlist that an LLM judge must admit | Judge score 0.85 (INTENT_THRESHOLD) |
Four ways a semantic cache goes wrong
1. The threshold trade-off. A single cut-off has to separate real paraphrases from near misses that share most of their words. The vCache paper (ICLR 2026) found that a static threshold gives either a low hit rate or a high error rate, and proposed learning a threshold per cached entry. Questions that differ by a negation, an entity or a version number are the usual casualties.
2. Agent traffic. LiteLLM's documentation warns that its semantic cache is designed for single-shot prompts. In agent loops, consecutive turns embed at around 0.99 similarity, raising the threshold does not reliably fix it, and assistant tool calls are not part of the embedded text. LiteLLM recommends exact caching for agents.
3. Replayed tool calls. If a cached response contains a tool call, a hit replays that call, possibly with arguments meant for a different request. Bifrost skips caching tool-call responses by default for this reason. MorrowCache never stores them.
4. Freshness and scope. “What’s the latest Node release?” has the same meaning next month and a different answer. Personal questions have the same wording for every user and a different answer for each. A cache needs time limits and tenant boundaries, not only a similarity score.
How to measure a semantic cache
Hit rate is the number vendors lead with. The number that matters first is the false-hit rate.
- Build a labelled set of prompt pairs: real paraphrases, and traps that look alike but are not.
- Report precision, recall and the false-positive rate on the traps, not only overall hits.
- Replay a slice of real traffic with the cache in shadow mode before serving hits.
- Log every hit with the stored prompt it matched, so a wrong hit can be traced.
Using an LLM judge instead of a threshold
One answer to the threshold problem is to let similarity only propose candidates and have a model decide. The Krites paper (arXiv 2602.13165) uses an asynchronous LLM judge for near-threshold neighbours and reports up to 3.9 times more curated hits. MorrowCache puts the judge on the request path: a shortlist proposes up to five candidates, the judge admits or rejects, and if the judge errors or times out the request goes to the model.
The cost is a judge call on each candidate check, about $0.00002 per request in MorrowCache’s recorded sessions with the default cloud judge, and a semantic hit takes roughly 350 to 450 ms instead of the few milliseconds of a vector lookup. The judge can also run locally. Whether that trade is worth it depends on how expensive a wrong answer is for you.
Compare specific tools: GPTCache, LiteLLM caching, Redis LangCache and RedisVL, and gateway caches such as Helicone, Portkey and Cloudflare AI Gateway.
FAQ
- What is a semantic cache for LLMs?
- A cache that returns a stored model answer when a new prompt means the same thing as an earlier one, even if the words differ. An exact cache only hits when the request bytes are identical.
- Why do semantic caches return wrong answers?
- Most decide a hit with one embedding-similarity threshold. Questions that share most of their words but differ in a key detail, such as a negation, an entity or a date, can score above the threshold. A threshold low enough to catch real paraphrases also admits some of these near misses.
- Is semantic caching safe for AI agents?
- Only with guards. Consecutive agent turns embed almost identically, and replaying a stored tool call can run an action with the wrong arguments. Scope hits to the same prior conversation, never replay tool calls, and prefer a miss when unsure.
- How should I measure a semantic cache?
- Track the false-hit rate first, then the hit rate. A cache with a high hit rate and a few percent wrong answers can cost more in bad output than it saves in tokens.