Accuracy: does it admit the wrong question?
The eval set is labelled prompt pairs: real paraphrases that should hit, and look-alikes that should not, such as negations and different entities. We compare the judge with a word-overlap matcher (Jaccard similarity at 0.35), a common cheap baseline.
| Matcher | Pairs | False-positive rate | Precision | Recall |
|---|---|---|---|---|
| LLM judge (Jev, live, admit-v1) | 100 | 0.00 (0 of 50 traps) | 1.00 | 0.38 (19 of 50) |
| Word overlap, Jaccard 0.35 | 100 | 0.48 | 0.33 | 0.24 |
| Word overlap, Jaccard 0.35 | 103 | 0.49 | 0.32 | 0.24 |
- The live judge run was on 19 September 2026 against the admit-v1 gate, and cost $0.0017 for 100 pairs.
- The current 103-pair set adds negation and entity traps. Its live judge numbers have not been re-run yet, so we do not quote them.
- Recall of 0.38 means the judge let 31 of 50 real paraphrases miss. That is the price of preferring a miss to a wrong hit: those requests go to the model as usual.
- Zero false positives on 50 traps is not a guarantee of zero in production.
Latency and cost: three scripted sessions
Each session sent a fixed list of questions and their paraphrases through the proxy with the default cloud judge and real upstream calls via OpenRouter. These are scripted demos, not production traffic, so the hit rate reflects how many paraphrases the script contained.
| Session | Requests | Hits | Median hit | Median miss | Judge cost |
|---|---|---|---|---|---|
| Broad | 20 | 8 | 399 ms | 4,328 ms | $0.00039 |
| Raw | 17 | 8 | 389 ms | 2,702 ms | $0.00031 |
| Tight | 12 | 9 | 369 ms | 5,722 ms | $0.00020 |
Judge cost works out to roughly $0.00002 per request. With a cheap upstream model, that can be a real share of the saving; on the broad session the judge cost $0.00039 against $0.00128 saved. The saving grows with the price of the upstream model and the length of its answers. A local judge removes the judge cost.
Reproduce it
git clone https://github.com/kushals256/jevcache && cd jevcache && npm install
npm run eval # offline, Jaccard baseline
LIVE=1 OPENROUTER_API_KEY=sk-or-... npm run eval # adds the live judgeNot measured yet
- Hit rate on real coding-agent sessions over days.
- False hits in production traffic.
- Local judges (Kev, Laya) on the same eval set.
- Throughput under concurrency. The cache is one SQLite process.
If you run MorrowCache on real traffic, the /stats page reports hit rate, latency percentiles and estimated savings. Sharing those numbers in a GitHub issue helps more than anything else.