gallery unit · throughput & caching
Silent cache-miss guardrail
The cache returns the right answer every time. It also never hits, so every request quietly pays full price and no test says a word.
scenario sandbox
You are the platform engineer. The same warm workload, 240 requests over 12 hot prompts, runs against whichever cache-key config shipped. Live in your browser, deterministic every run.
pick the cache-key config
flagged by the guardrail only
The guardrail flagged this config: hit ratio 0% on a workload built to repeat. 240 of 240 requests paid full cost.
Every answer was still correct, so the functional test saw nothing wrong.
| req | cache key | result |
|---|---|---|
| #0 | 0:SYSTEM: answer from context. DOC… | miss |
| #1 | 1:SYSTEM: answer from context. DOC… | miss |
| #2 | 2:SYSTEM: answer from context. DOC… | miss |
| #3 | 3:SYSTEM: answer from context. DOC… | miss |
| #4 | 4:SYSTEM: answer from context. DOC… | miss |
| #5 | 5:SYSTEM: answer from context. DOC… | miss |
| #6 | 6:SYSTEM: answer from context. DOC… | miss |
| #7 | 7:SYSTEM: answer from context. DOC… | miss |
hit ratio 0% · 240 of 240 requests paid full cost
every returned answer matches the direct computation, so the test people trust sees nothing wrong.
The workload, cache, and both checks are live and deterministic, the same logic as the Python run, verified by web/parity-check.mts (PARITY OK, identical hit ratios).
what you are looking at
01 · data in
A warm stream of 240 requests over 12 hot prompts, the same every run.
02 · check runs
The cache runs with the key config you picked. The guardrail checks the hit ratio against a floor; a functional test checks every answer.
03 · verdict out
The banner reports the hit ratio, what it cost, and which check fired.
Go deeper
Watch the walkthrough (45 s, silent)
Four cache keys, one honest hit rate · recorded from this page, unedited.
The measured result: 95% vs 0% hits
A cache with a key bug still returns the right answer. It just never hits, and nothing tells you.
Same warm workload, four key configs
240 requests · 12 hot prompts · guardrail ~0.001 ms
| Check | Silent-miss configs | Correct config | Verdict |
|---|---|---|---|
| Silent-cache-miss guardrail | FLAGGED 3/3 | passes | catches it |
| Functional correctness test | passes 0/3 | passes | misses it |
A working cache serves 95% of requests from memory, 12 real calls. Each buggy config drops to 0%, 240 real calls, 20× the compute, while returning identical, correct answers. The functional test passes all of them. Only the guardrail sees the cost.
Where the problem statement comes from
“Shorter prompts cannot be cached, even if marked with cache_control. Any requests to cache fewer than this number of tokens will be processed without caching, and no error is returned.”
Anthropic prompt-caching docs · Minimum cacheable prompt length · accessed 2026-08-04
The full mechanism: why the functional test can't see it
- The answer is computed from the query, not the cache. A wrong cache key never produces a wrong result; on a miss the system recomputes correctly. So a test that checks outputs passes whether the cache hit or not.
- The failure is silent by design. Below the minimum length the provider declines to store and returns no error; a volatile field in the prefix breaks exact matching so the key is never identical twice. Either way: correct output, full cost, no signal.
Correctness cannot see cost. The guardrail asserts a different property: on a warm workload, one where requests are known to recur, the cache-hit ratio must clear a floor derived from the repetition structure. A cache that never hits reads 0.00 and is flagged, in about a microsecond.
bug: cache key = f"{timestamp}:{prompt}" -> key differs every request -> 0% hits ✗
fix: cache key = canonical(prompt) -> same key for same prompt -> 95% hits ✓
guardrail: on a warm workload, assert hit_ratio >= floorEvidence + how to reproduce
One warm workload run through four key configs, scored by the guardrail and by a real functional correctness test. The functional test is the sharp part: the thing teams trust to validate a cache passes every silently broken config. Anchored on Python’s own functools.lru_cache, a real off-the-shelf cache, whose cache_info() shows the identical miss (0 hits / 240 on the volatile key). Synthetic workload; the repetition is known by construction, stated on the page.
make setup && make test && make run # $0, laptop, no GPU, no network
Honest gaps
The cache models the two documented provider rules (exact-prefix match, minimum cacheable length) rather than integrating with a live provider. The guardrail catches total silence, 0%, cleanly; partial degradation (say 95% down to 60% after a config change) needs a tuned floor or trend tracking, not a fixed one. And the workload’s repetition is known here by construction; a production stream’s warmth has to be measured before the floor means anything.