At a glance
| LiteLLM proxy caching | MorrowCache | |
|---|---|---|
| Product | Gateway with caching as a feature | Response cache only |
| Cache backends | Exact: Redis, S3, GCS, local, disk. Semantic: Redis, Valkey or Qdrant | SQLite, one process |
| Semantic hit | Embedding cosine similarity; docs examples use 0.8 | Shortlist proposes, an LLM judge admits |
| Endpoints | Chat Completions, Responses, Anthropic Messages and more | Chat Completions only; /v1/responses returns 501 |
| Agent traffic | Docs recommend exact caching, not semantic | Turn cache scoped to the same prior conversation |
| Tool calls | Not part of the semantic embedding, per its docs | Responses with tool_calls are never stored |
| Cache scope | Shared across end users unless semantic_cache_scope: end_user | X-Jevcache-Tenant isolates caches |
| Hit header | x-litellm-semantic-similarity | X-Jevcache, X-Jevcache-Tier, X-Jevcache-Reason |
The agent problem, in LiteLLM’s words
LiteLLM’s semantic caching page says the feature is designed for single-shot prompts and goes badly wrong on agentic traffic: consecutive turns of a conversation embed at around 0.99 similarity, raising the threshold does not reliably fix it, and assistant tool calls are not included in the text that gets embedded. It recommends exact caching for agents.
MorrowCache was built for that case. A turn can only hit against entries with the same prior conversation; only the latest ask is judged for intent; and a response that contains tool calls is returned but never stored, so a tool step always runs.
Where LiteLLM wins
- It is a full gateway: 100+ providers, fallbacks, virtual keys, budgets and spend tracking.
- Its caches can live in Redis or Qdrant and serve many proxy replicas. MorrowCache is one SQLite process.
- It covers the Responses API and Anthropic Messages, which MorrowCache does not.
- A semantic hit is a vector lookup, with no judge call.
Using both
LiteLLM’s proxy speaks the OpenAI API, so it can be MorrowCache’s upstream. Your app talks to MorrowCache; misses go to LiteLLM, which routes them.
UPSTREAM_BASE_URL=http://127.0.0.1:4000/v1 # your LiteLLM proxy
UPSTREAM_API_KEY=sk-litellm-virtual-key
OPENROUTER_API_KEY=sk-or-... # for the default cloud judgeVerdict
If you need a gateway, use LiteLLM, and keep its cache on exact mode for agents as its docs advise. If you want paraphrase and retry hits for agents with a judge in the loop, put MorrowCache in front. See also Helicone, Portkey and Cloudflare caching.
FAQ
- Can I use MorrowCache with LiteLLM?
- Yes. LiteLLM's proxy exposes an OpenAI-compatible API, so you can set MorrowCache's UPSTREAM_BASE_URL to your LiteLLM proxy and keep LiteLLM's routing, budgets and keys behind the cache.
- Does LiteLLM semantic caching work for agents?
- LiteLLM's own documentation says its semantic cache is designed for single-shot prompts and recommends exact caching for agentic traffic, because consecutive turns embed almost identically.