Extending the conversation history just to reach Anthropic’s minimum prompt caching threshold can cost more than keeping a shorter context window without caching. That’s what we found when optimizing a voice assistant whose responses rarely required more than a few recent exchanges.
Key Takeaways:
- With Claude Haiku 4.5, the first cache write happens only after the prompt reaches 4,096 tokens. For our voice assistant, getting there meant keeping more conversation history than the replies typically needed.
- The stepped window saved money on cache hits but lost those savings whenever trimming forced a rewrite of the remaining history.
- A fixed six-pair window without caching had the lowest input cost in our comparison. Of the 287 learner turns we evaluated, 94.1% needed no more than two earlier exchanges.
The Setup
The application is a voice assistant for practicing English. The learner speaks, speech-to-text transcribes it, an LLM generates a short spoken reply, and text-to-speech voices it. The model is stateless: the orchestrator keeps the session history itself and resends the entire conversation with every new message.
That is exactly what makes caching attractive. The same history is sent over the wire repeatedly, byte for byte identical to the corresponding prefix of the previous request. Paying the full input token price for it every time is obviously wasteful.
Anthropic has three input token prices: a regular token costs 1.0×, a cache write costs 1.25× (for the five-minute option), and a cache read costs 0.1×. Writing to the cache costs 25% more once, while every subsequent read costs one tenth of the regular price. The break-even point is the first reuse.
There are two cache storage options: five minutes and one hour, with the one-hour option costing twice as much for the cache write. Since every cache hit resets the expiration timer, the five-minute option is enough for a real-time voice conversation. Messages typically arrive less than a minute apart. Each one refreshes the cache lifetime, and the cache should remain active for as long as the learner keeps talking.
The recorded sessions confirm this. Across 73 consecutive turns, the median gap between requests was 40 seconds. So the one-hour option would simply be unnecessary.
The initial assumptions therefore seemed sound: an identical prefix, a break-even point after just one reuse, and a request cadence that keeps the cache active throughout the conversation.
On paper, this looks almost like free savings. In practice, it costs us more than doing nothing.
Two Conditions for Prompt Caching
The first is a minimum length. Caching does not apply to prompts below a per-model threshold:
| Model | Minimum cacheable prompt |
| Claude Haiku 4.5 | 4,096 tokens |
| Claude Sonnet 5 / 4.6 | 1,024 tokens |
| Claude Opus 4.8 | 1,024 tokens |
| Claude Opus 5 | 512 tokens |
The documentation is explicit about what happens below this threshold:
- Shorter prompts cannot be cached, even if marked with
cache_control. Any requests to cache fewer than this number of tokens will be processed without caching, and no error is returned.
Nothing else in the response signals the omission. The request succeeds, the answer is normal, and the tokens are billed at the regular input rate. The only way to detect it is to read the usage fields: if cache_creation_input_tokens and cache_read_input_tokens are both zero, nothing was cached.
The second is exact prefix matching. A cache entry is keyed on the literal prefix of the request. Any change to the beginning of the conversation invalidates everything after it.
This is the condition that turns a sensible-looking optimization into a costly option. A sliding window that keeps the last N exchanges moves the first message forward on every single turn. The prefix never matches. The request pays 1.25× to write a new entry and then throws it away one turn later, having read nothing. Trimming the history on every turn while caching is strictly worse than either trimming or caching alone in this scenario.
What the Logs Showed
All measurements below come from Claude Haiku 4.5, accessed through a LiteLLM gateway.
Three policies were run against the live pipeline, selected using a flag in the orchestrator: no limit, no cache (full history resent, no cache_control anywhere); no limit, cache on (full history resent with a cache breakpoint on the last message); and a stepped window, which caches the same way but cuts the history back toward 4,600 tokens once it exceeds 6,100. The history is trimmed in discrete steps: nothing is dropped between cuts, so the prefix stays byte-identical.
Every call logs two lines: an estimate of the context before the request and the billed usage after it.
The stepped-window run, around turn 22. Before its first cut, this policy resends everything, exactly like no limit:
[stream2] context: policy=A_budget_floor turns=21 sys_tok~82 hist_tok~4045 user_tok~110
[stream2] usage: policy=A_budget_floor input=4038 cache_write=0 cache_read=0 total_input=4038 cache_hit=0%
[stream2] context: policy=A_budget_floor turns=22 sys_tok~82 hist_tok~4227 user_tok~89
[stream2] usage: policy=A_budget_floor input=3 cache_write=4186 cache_read=0 total_input=4189 cache_hit=0%
[stream2] context: policy=A_budget_floor turns=23 sys_tok~82 hist_tok~4404 user_tok~86
[stream2] usage: policy=A_budget_floor input=3 cache_write=179 cache_read=4186 total_input=4368 cache_hit=96%At 4,038 tokens, both cache counters are zero. At 4,189, the whole prompt is written to the cache, and on the next turn it returns as a cache read while only the new exchange is written.
The no limit, cache on run was recorded on a different day with a different learner. The cache=on flag was set on all forty-one preceding calls as well, and none of them cached anything:
[stream] context: turns=41 sys_tok~82 hist_tok~4230 user_tok~23 cache=on
[stream] usage: input=4087 cache_write=0 cache_read=0 total_input=4087 cache_hit=0%
[stream] context: turns=42 sys_tok~82 hist_tok~4305 user_tok~39 cache=on
[stream] usage: input=3 cache_write=4167 cache_read=0 total_input=4170 cache_hit=0%Two independent conversations place the caching threshold between 4,087–4,170 tokens.
The stepped window at its first cut. Four consecutive calls: working, working, cut, working again.
[stream2] context: policy=A_budget_floor turns=30 sys_tok~82 hist_tok~5630 user_tok~186
[stream2] usage: policy=A_budget_floor input=3 cache_write=259 cache_read=5409 total_input=5671 cache_hit=95%
[stream2] context: policy=A_budget_floor turns=31 sys_tok~82 hist_tok~5904 user_tok~278
[stream2] usage: policy=A_budget_floor input=3 cache_write=332 cache_read=5668 total_input=6003 cache_hit=94%
[stream2] context: policy=A_budget_floor turns=24 sys_tok~82 hist_tok~4669 user_tok~65
[stream2] usage: policy=A_budget_floor input=3 cache_write=4629 cache_read=0 total_input=4632 cache_hit=0%
[stream2] context: policy=A_budget_floor turns=25 sys_tok~82 hist_tok~4796 user_tok~156
[stream2] usage: policy=A_budget_floor input=3 cache_write=238 cache_read=4629 total_input=4870 cache_hit=95%At turn 31, the history is 5,904 tokens, under the bound, and 94% of the request is served from cache. Adding that exchange pushes it past 6,100, so seven pairs are dropped from the front: turns falls from 31 to 24, the history from 5,904 to 4,669. The request is still 4,632 tokens and still above the minimum, but its first message has changed, so cache_read is zero and the whole window is rewritten. One turn later, the new prefix is established and the hit rate returns to 95%.
What That Costs
The behavior above, priced. We compared four policies on the same learner profile where the system prompt is 82 tokens, each exchange is 190, and the current turn is 124, using Haiku 4.5 input pricing. Only reply-call input costs are included; output costs are assumed to be identical across policies and are excluded. The fourth policy is a fixed six-pair window with no caching at all; where the number six comes from is the subject of a later section.
| Policy | 20 turns | 40 turns | 100 turns |
| No limit, no cache | $0.0402 | $0.1564 | $0.9611 |
| No limit, cache on | $0.0402 | $0.0642 | $0.1578 |
| Stepped window 4,600–6,100 | $0.0402 | $0.0679 | $0.1530 |
| Fixed 6-pair window, no cache | $0.0229 | $0.0498 | $0.1306 |
The first three rows are identical in the 20-turn column because the context remains below both the 4,096-token threshold and the trimming bound. For this learner profile, the threshold is crossed on turn 22, which is why the policies diverge only in the 40- and 100-turn columns.
The figures below show the cost of each request in ordinary input-token equivalents, accounting for cache reads at 0.1× and cache writes at 1.25× the standard input-token price. For uncached requests, the physical token count and the cost-equivalent count are the same. For cached requests, they differ. At turn 40, for example, the cost equivalent of 980 tokens comes from 7,426 cache-read tokens and 190 cache-write tokens, even though the request contains 7,616 tokens in total.
Per-turn input cost (input-token equivalents):
| Policy | Turn 10 | Turn 40 | Turn 100 |
| No limit, no cache | 1,916 | 7,616 | 19,016 |
| No limit, cache on | 1,916 | 980 | 2,120 |
| Fixed 6-pair window | 1,346 | 1,346 | 1,346 |
Without a context limit, each turn adds 190 input tokens to the cost of the previous turn. Caching reduces that considerably: at turn 40, the cost drops from 7,616 to 980 input-token equivalents. But the savings do not last indefinitely. The cached prefix keeps growing, and even at one-tenth of the standard input-token price, its cost continues to rise. By turn 100, a cached turn costs more than an uncached turn did at turn 10. The fixed window is the only policy that keeps the per-turn cost constant.
The stepped window was the attempt to combine both approaches, and the log above is the reason it fails. Its cached turns are genuinely cheap, around 770 tokens, but every cut rewrites the whole buffer at 1.25×: 5,750 instead of 770. With the context growing by 190 tokens per turn inside a 1,500-token step, that happens every eight turns. Amortized, the rewrites add approximately 620 token-equivalent units per turn. Between turns 40 and 100, the stepped policy averages 1,419 tokens per turn against 1,346 for the fixed window: close enough to call it a tie. And a tie is not good enough because reaching the threshold at turn 22 has already cost 49,471 tokens compared with the fixed window’s 25,622. A deficit of roughly 24,000 tokens is not repaid by drawing level.
The 100-turn column is extrapolation. The longest session in the measured set is 46 turns.
Turns Are the Wrong Unit
The natural way to express a history limit is “keep the last twenty exchanges.” That limit does not control token usage, because an exchange is not a fixed quantity. Across the recorded sessions, one exchange ranged from 80 to 188 tokens, depending on nothing but how talkative the learner was.
What the cache responds to is the size of the request in tokens, so that is what has to be modeled. A request has three parts: the system prompt, which never changes; the accumulated history; and the current learner turn. The history is the only part that grows, and it grows by exactly one exchange per turn.
input(t) = system + (t - 1) × pair + current_turn
The request therefore grows linearly, and the slope of that line is the size of one exchange. Setting it equal to the 4,096-token minimum and solving for the turn number gives:
turn = ceil((4096 − system − current_turn) / pair) + 1
The ceiling matters: caching engages on the turn that crosses the boundary, not on the one before it.
An exchange is the learner’s turn plus the tutor’s answer, and most of the variation comes from the first. Across these sessions, the tutor answered with 60, 62, and 60 tokens while learners averaged 31, 37, and 124, so the second term can be treated as a constant of about 66 tokens, counting the few structural tokens the API adds per message. That makes pair ≈ learner turn + 66.
Applied across a range of learner profiles, with the 82-token system prompt of our reply call:
| Mean learner turn (tokens) | Turns to reach 4,096 |
| 40 | 39 |
| 60 | 33 |
| 124 | 22 |
| 200 | 16 |
| 300 | 12 |
Across 64 calls where both our estimate and the billed figure were logged, the measured ratio was 4.18, tightly clustered between 4.06 and 4.30. The figure covers our own pipeline, including structural tokens counted by the API but excluded from our character count. Dividing by four overestimates the billed count by about 4.5%, which is the safe direction for a budget check, and precise enough to read a token count back as words and minutes.
This estimate is enough to reason about how much history is needed to reach the cache threshold. Even the most talkative profile needs twelveturns of accumulated history before the cache can engage. A single utterance never reaches the cache threshold: that would take roughly 4,000 tokens in one message, about 3,000 words, or 25 minutes of uninterrupted speech, and voice activity detection cuts an utterance at the first pause. The cache is reachable only through accumulation. Learner turns account for most of the variation, which is why a limit has to be expressed in tokens rather than in a number of turns.
How Much History the Reply Actually Needs
Cost analysis narrows the window. It cannot tell you how small is too small. For that, we measured the conversations directly.
For every learner turn in the recorded sessions, a judge model was shown the preceding exchanges, each labeled by distance rather than by absolute position, so the judge reads a label instead of doing arithmetic, and asked for the furthest-back exchange still required to answer correctly.
EARLIER EXCHANGES:
[-3] learner: I used to live in Colorado, I grew up there
[-3] tutor: That sounds lovely, what brought you back?
[-2] learner: My family is all still there
[-2] tutor: Do you visit often?
[-1] learner: Once a year maybe
[-1] tutor: Any plans for this year?
LATEST LEARNER MESSAGE:
Actually we're thinking about the Grand CanyonThe prompt carried only the criterion: an exchange counts as required if, without it, the tutor would fail to resolve a reference, repeat an answered question, or contradict a stated fact. Topical relevance does not count, and without that restriction, the judge answers “all of it” nearly every time. The response shape was enforced with a forced tool call rather than requested in prose.
Across 287 turns from 11 sessions:
- 42.5% of learner turns need no prior context at all
- 87.1% need at most one exchange back
- 94.1% need at most two
The distribution has a clean break. Depths 0 through 3 are dense; there is nothing at depth 4; and beyond that, only isolated singletons at 5, 7, 9, 11, 13 and 18. Ordinary conversation lives within three exchanges, and what lies further back is not a gradient but occasional long-range callbacks.
Going from six pairs to sixteen costs 2.4× and buys 1.4 percentage points — four turns out of 287.
TL;DR
Anthropic prompt caching isn’t always the cheapest option for conversational AI applications. For a voice assistant using Claude Haiku 4.5, extending the conversation history to reach the 4,096-token caching threshold costs more than maintaining a fixed six-pair window without caching.
The shorter window was also sufficient for most recorded conversations: 94.1% of 287 learner turns needed no more than two preceding exchanges. The results suggest that context length should be based on what the application actually needs, not on the minimum required to enable caching.
