What each gateway’s cache does
| Gateway | Match type | Notes from the docs |
|---|---|---|
| Helicone | Exact only | Opt in with Helicone-Cache-Enabled. TTL defaults to 7 days. Buckets can hold up to 20 responses, picked at random. |
| Cloudflare AI Gateway | Exact only | SHA-256 of provider, endpoint, model, auth and body. Off by default. Semantic search listed as planned. |
| Portkey | Exact; semantic on select Enterprise plans | Semantic uses cosine similarity, default 0.95, after an exact miss. Ignores the system prompt; limited to 4 messages and 8,191 tokens. |
| Bifrost | Exact, then semantic | Cosine default 0.8, TTL 5 minutes, needs a vector store and a cache key header. Skips conversations over 3 messages and tool-call responses by default. |
| Kong AI Gateway | Semantic (Enterprise plugin) | Redis, Valkey or pgvector. Embeds the last N messages; can stream cached responses. |
| TensorZero | Exact only | Stored in ClickHouse; default mode is write-only. |
| MorrowCache | Exact, then judged intent | LLM judge admits; same-prior turn cache for agents; tool-call responses never stored; freshness refusals. |
When an exact gateway cache is enough
Exact caches cannot return a wrong answer for a different question, which is why Helicone, Cloudflare and TensorZero stop there. If your prompts are generated from templates and repeat byte for byte, use the cache your gateway already has. A semantic layer only pays off when the same question arrives in different words, which is common with people typing and with agents retrying.
Where MorrowCache fits
- In front of a gateway: set MorrowCache’s UPSTREAM_BASE_URL to the gateway’s OpenAI-compatible endpoint.
- Without a gateway: point your client straight at MorrowCache and let it call OpenRouter, OpenAI or a local model.
- Free and local: no plan tier, and the judge can run on your machine.
MorrowCache is not a gateway. It has no provider routing, budgets or dashboards beyond its /stats page, and it handles Chat Completions only. See LiteLLM for a gateway comparison with configuration.