Prompt Caching Explained: How to Cut LLM Cost by up to 90% Without Losing Quality

Short answer: Prompt caching stores the model's precomputed attention state (the KV matrices) for a prompt prefix so it never has to recompute those tokens on the next request. Reusing that state is dramatically cheaper and faster: Anthropic reports cost cuts of up to 90% and latency cuts of up to 85%, with cache reads billed at $0.30/M tokens versus $3.00/M for fresh input. The output is identical to an uncached run.

What prompt caching actually is

When an LLM processes your prompt, it computes a key/value (KV) matrix for every token in the attention layers. That prefill step is the expensive part. Prompt caching keeps those KV matrices around after a request, so if your next request starts with the exact same prefix, the provider skips prefill for those tokens and jumps straight to generating.

Crucially, it is a compute shortcut, not a different model. Same weights, same sampling, same result. You only save money and time on the tokens you reused.

How the three big providers differ

The key contrast: OpenAI caches automatically and gives you a 50% discount with zero code changes, while Anthropic requires you to mark cacheable blocks with cache_control but rewards that effort with a much steeper discount. Across providers the KV matrices are typically held for roughly 5–10 minutes after a request (as documented by PromptHub and ngrok), so caching pays off for bursty, repeated traffic, not one-off calls hours apart.

Design your prompt for cache hits

The rule is simple: static content first, variable content last. A cache hit requires the prefix to be byte-for-byte identical, so anything that changes per request must come after everything you want cached.

• Put the system prompt, tool/function definitions, and long reference docs at the very top.
• Put the user's turn, timestamps, retrieved snippets that change, and IDs at the bottom.
• On Anthropic, place cache_control at the end of the last static block so everything above it is cached.

# Anthropic: cache the big static prefix, vary only the user turn
messages = [{
    "role": "user",
    "content": [
        {"type": "text", "text": SYSTEM_DOCS,          # static, huge
         "cache_control": {"type": "ephemeral"}},       # cache up to here
        {"type": "text", "text": user_question},        # dynamic, cheap
    ],
}]

Pitfalls that quietly kill your savings

Any prefix change invalidates the cache. Injecting the current time, a random request ID, or reordering tool definitions at the top of the prompt means a full recompute every time. Move all of it below the cached boundary.

Agentic loops break the cache in a subtle way. Each tool call appends a result to the conversation, which changes the suffix but not the prefix, so the original static prefix still hits. The trap is mutating earlier messages, re-summarizing history in place, or re-ordering tool outputs, which shifts the prefix and forces recomputation on every loop iteration. Append-only conversation history is what keeps long agent runs cheap.

The TTL is short. With a ~5–10 minute window, low-traffic endpoints may never see a second hit before the entry expires. Caching helps most under sustained or bursty load; measure your hit rate before assuming the discount.

You still pay to write the cache. On Anthropic, the initial cache write costs more than a normal token (roughly 1.25x), so caching a prefix used only once is a net loss. It wins when the same prefix is reused many times.

Cutting cost up to 90% comes down to one discipline: keep a large, stable prefix and append everything that varies — never edit what came before.

Related: more writing.