What prompt caching does
When many requests start with the same long prefix, such as a system prompt, tool definitions or a block of repo context, the provider can keep the processed prefix and reuse it. OpenAI applies this automatically once the prefix reaches 1,024 tokens. Anthropic lets you mark breakpoints with cache_control.
The model still runs. OpenAI’s documentation says prompt caching does not change how the model generates output tokens, and Anthropic’s says output is generated fresh each time. You pay less for the cached input; output tokens cost the same.
What response caching does
A response cache sits in front of the model. If a request matches one it has seen, exactly or by meaning, it returns the stored answer and the model is never called. There are no input or output tokens to pay for, and the answer arrives in milliseconds to a few hundred milliseconds instead of seconds.
| Prompt caching | Response caching | |
|---|---|---|
| Who runs it | The model provider | You, in your app or a proxy |
| Does the model run? | Yes | No, on a hit |
| Saves on | Input tokens of a shared prefix | The whole call: input, output and latency |
| Helps when | Many calls share a long prefix | The same question or job repeats |
| Risk | None to correctness | A wrong hit returns the wrong answer |
| Examples | OpenAI, Anthropic, Vercel AI Gateway caching: 'auto' | MorrowCache, GPTCache, LiteLLM cache, Helicone |
Using both
A coding agent is a good example. Every turn carries the same large system prompt and repo context, so prompt caching cuts the input cost of every call. When the agent retries the same step with a reworded instruction, a response cache can return the final answer without a call at all. MorrowCache does the second part for any OpenAI-compatible client. Calls that miss still go to the provider, so its prompt caching applies to them as usual.
If you are choosing a response cache, read semantic caching and where it goes wrong first.
FAQ
- Is OpenAI prompt caching the same as caching responses?
- No. Prompt caching reuses the provider's computation for a repeated input prefix, so input tokens are cheaper. The model still generates a new answer and you still pay for every output token. A response cache skips the model call and returns a stored answer.
- Can I use prompt caching and a response cache together?
- Yes. They work at different layers. A response cache in front of the model removes repeat calls; prompt caching makes the calls that still go through cheaper when they share a long prefix.
- Which saves more money?
- It depends on the traffic. Prompt caching helps every call with a long shared prefix, such as a big system prompt or repo context. A response cache helps only when the same question or the same job repeats, but then it saves the whole call, output tokens and latency included.