At Induction AI, we’re building context optimization to make AI agents more efficient. Our service reduces inference costs while preserving task accuracy without requiring changes to the agents themselves.
To accomplish this, we spend a lot of time understanding the behaviors of the caching systems in the models.
Recently, Gemini 4 Argon was released and we were curious to see how the caching behavior had changed from previous models. In particular, we saw many quirks with caching in Gemini 3.1 Pro Preview and were hoping Gemini 4 would be more predictable. The more predictable it is, the better we can optimize our context management layer.
Fortunately, Gemini 4 Argon does seem to be more predictable, though it is not without its own quirks.
The basics
The prompt cache works by re-using what has already been sent before. If you only ever append to a thread, you will get a discount on the tokens that the provider has seen before. Like the recent changes to OpenAI and Anthropic, this discount is significant: 95% off of the normal token price. So it is pretty important to get right.
We tested the caching behavior by appending to a thread to see where it starts hitting. On Argon, it worked right at the documented 4096 token threshold. That is to say if you are sending 4000 tokens on step 1, there will be no caching on the next turn. However, if you send 4100 tokens, there will be caching on the next turn.
- cache read
- billed full price
- breakpoint
Turn 1 is 4000 tokens
at $2/M full price, $0.10/M cached:turn 1 0.80¢turn 2 1.30¢total 2.10¢
Turn 1 is 4100 tokens
at $2/M full price, $0.10/M cached:turn 1 0.82¢turn 2 0.54¢total 1.36¢
Turn 2 appends 2500 tokens in both conversations.
This is great to see because the 3.1 Pro Preview behavior varied wildly. In our tests, Gemini 3.1 Pro Preview was only dependable after 9000 tokens.
The 2049 quirk
Like the newer models from OpenAI and Anthropic, the caching behavior seems to operate on the notion of breakpoints. This means that there are explicit points on each request and that is where it can re-use data on subsequent requests. The implicit breakpoint on OpenAI, for example, is basically at the end of each request.
So what actually gets cached when we send 4100 tokens and then append the next messages with say, 1000 more tokens?
OpenAI does what we would expect: it caches the 4100 from the previous call and we would pay full price for the new 1000 tokens. On the next turn of 1500 tokens, the same things happens and you only pay for the new tokens.
- cache read
- billed full price
- breakpoint
at $2/M full price, $0.10/M cached:turn 1 0.82¢turn 2 0.24¢turn 3 0.35¢total 1.41¢
One breakpoint per request, at its end. Each turn caches all but the last 3 tokens of the previous request.
Gemini 4 seems to set its breakpoints differently. It sets it at the end of the request just like other providers. But it also sets another breakpoint 2049 tokens before the end of every request. This is new behavior from any model we have tested so far: counting backwards from the end instead of blocks from the beginning.
On top of that, there looks to be a requirement that we always pay for at least 2049 tokens. What this means is that, if the new content is only 1000, then we reach back into the previous request. In this case, it only hits the cache of 2051 (4100-2049) and we pay for 3049 (2049+1000).
This continues even on the next request. If we send 1500 more tokens, it will go back to the -2049 breakpoint from the previous turn. One thing we do not understand is why it did not use the original 4100 breakpoint in the thirs request. This suggests that it was removed when it was not immediately put into use or only the ones from the last turns are put into use.
- cache read
- billed full price
- breakpoint
- breakpoint it can’t use
- dropped
at $2/M full price, $0.10/M cached:turn 1 0.82¢turn 2 0.63¢turn 3 0.74¢total 2.19¢
Each request leaves breakpoints at its end and 2049 tokens before it, and can’t resume within its own last 2049 tokens. Turn 2 resumes at 2051, and turn 1’s end breakpoint is dropped, so turn 3 can only resume at 3051 and pays full price for 3549: 2049 of earlier turns again, plus 1500 new.
However, if the second message is more than 2049 tokens (2500), it will cache the entire first message. That means we pay for 2500 and 4100 is cached. If the next message does not hit the 2049 amount, it will again reach backwards into the previous request.
- cache read
- billed full price
- breakpoint
- breakpoint it can’t use
at $2/M full price, $0.10/M cached:turn 1 0.82¢turn 2 0.54¢turn 3 0.76¢total 2.12¢
With 2500 new tokens, turn 2 resumes at the end of turn 1. Turn 3 adds only 1500, so it falls back to turn 2’s earlier breakpoint (4551) and pays full price for 3549.
See the code that shows this behavior here.
How this affects Gemini use
It is great to see some more predictable behavior, but this behavior makes Gemini 4 Argon pretty inefficient if the messages are short by their nature because there is a minimum of 2049 tokens that we have to pay for. If the messages consistently exceed that threshold, then it will work as expected.
This creates the counter-intuitive situation where there are cases where it is better to send more tokens in order to hit these breakpoints.
At Induction, we do this kind of research to learn things like this and automatically make our customers more efficient. We will be looking at our context management layer and its benchmarks to verify that this does so.