A Guide to Prompt Caching

Prompt caching is the single highest-leverage cost optimization available on modern LLM APIs, and most teams either skip it entirely or implement it in a way that silently loses most of the benefit. The mechanics are simple once you see them clearly — the mistakes that erase the savings are almost always about prefix ordering, not the caching feature itself.

Quick answer

Prompt caching lets a provider store the computed attention state for the repeated part of your prompt — typically the system prompt and tool definitions — so that subsequent requests sharing that exact prefix skip recomputing it and get billed at a steep discount instead of full price. Claude and current-generation GPT-5.6 both discount cached reads by 90% (0.1x input price); Gemini 3.1 Pro’s explicit caching discounts by 75%. The catch that trips up most implementations: caching is strictly prefix-based, so anything that changes between calls — a timestamp, a user ID, session data — has to come after the stable content, never before or mixed into it, or the cache never fires at all.

Key takeaways

• Caching works on exact-prefix matching, not similarity — a single character of difference in the “stable” portion breaks the cache for that request entirely.
• Both Claude and GPT-5.6 now charge a write premium (1.25x input) every time the cache needs refreshing, which changes the economics from what older guides describe — GPT-5.6’s writes were free before mid-2026, and no longer are.
• The most common cache-breaking mistake is putting dynamic content — a timestamp, a user ID — before or inside the stable prefix instead of strictly after it.
• Agentic tools that fan out into many tool calls in a single turn can silently exceed a provider’s cache lookback window, causing a quiet, unexplained cost spike with no error message.

How prompt caching actually works

When a model processes your prompt, the most computationally expensive part is the first pass — converting your text into the internal key-value (KV) representations the attention mechanism uses to generate a response. Prompt caching stores those KV representations for a prompt’s leading tokens, so that a later request sharing the exact same prefix can skip that expensive first pass and reuse the stored computation for everything up to where the new content begins.

This is why the split between what goes in a system prompt and what goes in a user message matters so much for cost, not just for behavior — see our guide to system prompts for the full split. The system prompt is exactly the kind of content caching is built for: identical across every call, ideally large enough to be worth caching in the first place.

Provider mechanics, verified current

The three major providers implement caching with meaningfully different mechanics, and the differences affect your real savings, not just your code.

• Claude (Anthropic): Explicit cache_control breakpoints. Cache reads bill at 0.1x input (90% off). Writes cost 1.25x input for a 5-minute TTL, or more for a 1-hour TTL. Cached entries are held in memory only, not stored at rest, and are ZDR-eligible.
• GPT-5.6 (OpenAI): Automatic caching above a 1,024-token stable prefix, with explicit cache breakpoints now also available as an option. As of mid-2026, cache writes cost 1.25x input — previously free — with a 30-minute minimum TTL on explicit caches. Reads still discount 90%.
• Gemini 3.1 Pro (Google): Explicit context caching requires creating a cache object via the API before referencing it, with a roughly 32K-token minimum and storage billed separately by time. Cached reads discount 75%. Implicit, automatic caching also exists and can reach higher discounts, but isn’t guaranteed the way explicit caching is.

The practical upshot: Claude and GPT-5.6 now share essentially the same write-premium economics, which wasn’t true before mid-2026 — older guidance describing OpenAI’s caching as “free to write” is describing the prior GPT-5.x generation, not the current one. If you’re modeling your own savings, our Prompt Caching ROI Calculator uses these current figures directly.

A worked example

Consider a support tool built on Claude with a 4,000-token system prompt (product documentation and tool definitions), handling 2,000 requests a day, each with roughly 300 tokens of unique customer question and 400 tokens of response.

Without caching: every one of those 2,000 daily requests pays full input price for the entire 4,300-token prompt (system prompt plus the unique question) and full output price for the 400-token response. At Claude’s current rates — see our AI Model Cost Calculator for the full per-token breakdown — that runs to roughly $1,134 a month.

With caching: the 4,000-token system prompt is written to the cache once, then read at a 90% discount for the remaining requests that day. Only the 300 unique tokens per request and the 400-token response are billed at full price. The monthly total drops to roughly $486 — a 57% reduction, driven almost entirely by the system prompt no longer being paid for at full price on every single call.

Run your own numbers, including the specific model and traffic pattern, in the Prompt Caching ROI Calculator — the shape of the savings holds broadly, but the exact percentage depends heavily on how large your stable prefix is relative to the unique content and output on each call.

When caching doesn’t help much

Caching’s benefit scales with how large the stable, repeated portion of a prompt is relative to everything else. A few situations blunt the benefit meaningfully:

• Short system prompts. If the stable prefix is a few hundred tokens and the unique per-request content and output dominate the call, there’s simply not much to discount — the math in the worked example above depends on the system prompt being the larger share of the bill.
• Very sparse, bursty traffic. If requests arrive far enough apart that the cache routinely expires between calls, you’re paying the write premium on most requests with few reads to offset it.
• Output-heavy workloads. Since caching never discounts output tokens, a workload that generates long responses from a short prompt sees a much smaller overall bill reduction than one with a large prompt and a short response, even at the same discount rate on the input side.

None of these situations make caching harmful to implement — there’s no real downside beyond the engineering time — but they do mean the advertised 90% figure describes the discount on cached tokens specifically, not the reduction in your total bill.

The mistake that silently erases the savings

Because caching matches on exact prefix, the single most common implementation mistake is placing anything that changes — a timestamp, a user ID, session-specific data — before or inside the content meant to be cached, rather than strictly after it.

Breaks the cache every call: a system message that starts with "Current time: 2026-08-10 14:32:07. Static knowledge base: ..." — the timestamp changes every request, so the entire prefix is different every time, and the cache never has a chance to hit.

Preserves the cache: the same static knowledge base placed first, with the timestamp and any other per-call data moved into the user message instead of the system prompt — system: "Static knowledge base: ...", user: "Current time: 2026-08-10 14:32:07. [actual question]". The system prompt is now identical on every call, which is exactly what a cache needs to hit.

This single fix is responsible for most of the gap between teams who see the full advertised discount and teams who implement caching but see almost nothing change on their bill.

The agentic tooling trap

There’s a second, less obvious way caching quietly fails, specific to agentic tools that make many tool calls within a single turn — see our guide to how AI coding assistants work for the broader mechanics of that plan-edit-test loop. Claude’s cache system checks a limited number of blocks backward from each breakpoint — roughly 20 — looking for a matching prior cache write. A simple chat application adds one or two content blocks per turn and never approaches that limit. A tool-heavy agent turn is different: one assistant message fanning out into a dozen tool calls, each returning its own result block, can add more than 20 blocks in a single turn.

When that happens, the next request’s cache breakpoint looks back the standard distance, finds no matching prior write within range, and silently rewrites the entire prefix at full premium rates. There’s no error and no warning — just a bill that quietly grows several times over on the input side, with nothing in the logs pointing directly at the cause. The documented fix is to place an intermediate cache breakpoint roughly every 15 blocks in long agentic turns, so the lookback window always has a nearby anchor to find, rather than relying on a single breakpoint at the very start of a turn that grows arbitrarily long.

Verifying caching is actually working shouldn’t wait until the monthly bill arrives. Every major provider’s API response includes cache-specific usage fields alongside the standard token counts — separate figures for cache-write tokens, cache-read tokens, and regular input tokens on that specific call. Checking these fields on a handful of real requests immediately after deployment — not weeks later — confirms whether the cache is hitting as expected. Cache-read token counts near zero on repeat calls with an otherwise-identical prompt is the clearest signal that a prefix-ordering mistake is quietly in effect, well before it shows up as an unexplained line on a monthly bill.

Common mistakes

• Putting per-request data before the stable prefix. This is the single most common cache-breaking error — see the worked example above.
• Assuming GPT-5.6 caching is still free to write. That was true for earlier GPT-5.x models; it changed with GPT-5.6’s mid-2026 release, and modeling costs on the old assumption understates the real bill.
• Not accounting for the write premium in savings estimates. A cache that’s rewritten frequently — because traffic is sparse or bursty — pays the write premium often enough to meaningfully cut into the read-side discount.
• Missing intermediate breakpoints in long agentic turns. A single breakpoint at the start of a turn that grows past the provider’s lookback window silently stops caching without any visible error.

Advanced tips

Structure prompts for caching from the start, not as a retrofit. Ordering stable content first and per-request content last is nearly free to do when a prompt is first being written, and considerably more work to fix once a prompt template is already embedded across a codebase.

Match TTL choice to actual traffic pattern, not the longest available option. A 1-hour TTL costs more to write than a 5-minute one, and only pays for itself if requests are frequent enough that the cache would otherwise expire and rewrite multiple times within that hour — for genuinely sparse traffic, the shorter, cheaper TTL can win even though it sounds less efficient.

Check your actual token counts before assuming caching is worth implementing. Below the minimum prefix length providers require for cache eligibility — 1,024 tokens is the common threshold — there’s nothing to cache yet, and the implementation effort has no payoff until the stable prefix grows past that point. Our token counter can confirm where your actual prompt sits before you invest engineering time.

The takeaway

Prompt caching’s economics changed meaningfully in mid-2026 — GPT-5.6 adopted a write-premium model much closer to Claude’s, closing a gap that made OpenAI look categorically cheaper to cache against in older comparisons. What hasn’t changed is the single biggest lever for actually capturing the savings: strict prefix ordering, with everything stable first and everything per-request strictly after it. Teams that get that ordering right see close to the advertised discount. Teams that don’t often implement caching, see almost no change on their bill, and conclude — incorrectly — that caching wasn’t worth the engineering time. It’s worth the time investment for nearly any application with a stable, substantial system prompt; it just has to be ordered correctly, and verified against real usage data, to actually work.

FAQ

Does prompt caching ever discount output tokens?

No, on any provider. Caching only ever affects input-side cost — the model still has to generate every output token fresh on every request, regardless of how much of the input was cached. Any savings estimate that extrapolates the input discount to the output side overstates the real number.

Why did GPT-5.6’s caching economics change?

GPT-5.6 (the Sol, Terra, and Luna tiers) reached general availability in July 2026 with a substantially revised caching model compared to earlier GPT-5.x releases — explicit cache breakpoints became available as a developer option, and cache writes moved from free to a 1.25x input premium, bringing OpenAI’s economics much closer to Anthropic’s existing model. Cost models or calculators built against the older, free-write assumption will overstate GPT-5.6 savings until updated.

How do I know if my prompt is even long enough to benefit from caching?

The common minimum is around 1,024 tokens for the stable, cacheable portion of the prompt — see our guide to how tokens work if you’re not sure how your prompt’s word count translates to a token count. Below that threshold, there’s nothing meaningful to cache and the implementation effort has no real payoff yet. Most system prompts with substantial tool definitions or reference material clear this threshold easily; short, simple system prompts often don’t.

Is caching worth implementing for a low-traffic application?

It depends on how sparse “low traffic” actually is relative to the cache TTL. If requests arrive frequently enough that the cache is still warm from the previous request, caching wins even at modest volume. If requests are sparse enough that the cache typically expires between calls, you’re paying the write premium almost every time with few reads to offset it, and the net benefit shrinks or disappears.

Can I cache content that includes retrieved documents from a RAG pipeline?

Only the parts that are genuinely stable across calls — a fixed knowledge base or document set works well. Retrieved chunks that change per query, the way a typical RAG pipeline’s output does, defeat the cache the same way any other per-request content does if placed in the cacheable prefix. See our RAG Pipeline Cost Calculator for how caching interacts with the rest of a retrieval system’s cost structure.

Does prompt caching work the same way for a single long conversation as it does across separate API calls?

Yes, and this is one of caching’s most useful applications — a multi-turn conversation about the same large document or corpus can cache that content once, then reuse it across every follow-up question within the TTL window, rather than re-paying for the full document on every single turn of the conversation.

Does using caching change the quality or content of the model’s responses?

No. Caching is purely a billing and latency optimization — the model produces the same response it would have generated without caching, just faster and cheaper for the cached portion. It’s not a form of compression, summarization, or approximation that could alter what the model actually returns.