Prompt Caching: Mechanics, Provider Pricing, and Cache-Aware Routing

TL;DR

Prompt caching stores the key-value cache a model computed for the start of a prompt and reuses it when a later request begins with the same tokens. The repeated prefix skips prefill and bills at a cached-input rate, typically 0.1x the base input price. It works on exact prefix matches only, so one changed byte near the top of the prompt costs the whole discount.

Published September 1, 2026. Updated September 7, 2026 with measured hit rates from production coding agents, engine internals, and the failure modes filed as bugs against real clients. Provider rates were fetched from each vendor's pricing page; Morph rates come from the same pricing table that bills the API.

Prompt caching is the reuse of a model's KV cache for a prompt prefix it has already processed. A request that begins with the same tokens as an earlier request skips prefill for the matching prefix and bills those tokens at a cached-input rate. Every major provider now sells it: 0.1x base input on Anthropic (0.025x on Claude Fable 5.1), OpenAI GPT-5.6, Gemini 3.x, and Moonshot Kimi K3, 0.032x on DeepSeek V4 off-peak, and $0.20 against $1.00 per 1M on GLM-5.3 744B at Morph. The mechanics are the same everywhere. The write surcharges, minimum prefix sizes, and retention windows are not.

How Prompt Caching Works

A transformer answers a prompt in two phases. Prefill reads every prompt token and computes a key and a value vector per token per layer, the KV cache. Decode then produces output tokens one at a time, attending to that cache. Prefill is compute-bound: its cost grows with prompt length, and for a coding agent the prompt is the entire conversation so far. Decode is memory-bound and produces a few hundred tokens per turn. On the session above, prefill handles 1,252,500 tokens and decode handles 15,000. Prefill is where the money goes.

Prompt caching keeps the KV cache from prefill and reuses it. Three rules define what can be reused:

• Exact prefix match. Attention at position n depends on every token before n. The cache for a token is only valid if all preceding tokens are identical. A request therefore hits from token zero up to the first token that differs from the cached prefix, and never past it.
• Block alignment. Engines store the KV cache in fixed-size blocks and hash whole blocks. Morph caches on roughly 1k-token blocks. A matching prefix that ends mid-block reuses the complete blocks before it; the partial block is prefilled again. Very short prompts have no full block to reuse, which is why every provider publishes a minimum cacheable length.
• Locality. The cache is GPU memory (sometimes tiered to CPU or NVMe) on the worker that ran the prefill. A request only hits if it reaches that worker while the blocks are still resident. Retention is bounded either by a TTL or by LRU eviction under memory pressure.

Multi-turn agent loops are the ideal shape for this. Each turn re-sends the previous turns verbatim and appends one new message, so the entire prompt except the newest turn is a prefix the engine has already processed. The provider reports how much hit in the usage object; on Morph it is usage.prompt_tokens_details.cached_tokens, and the remainder of prompt_tokens bills at the input rate.

For the serving side of the same story, throughput, batching, and where the KV cache lives across GPU, CPU, and NVMe, see LLM inference. For the KV-cache memory techniques that make long prefixes cheap to hold, see grouped-query attention and FP8 quantization.

Inside the Engine: Block Hashing and Radix Trees

No vendor doc explains why the match has to be a prefix rather than a diff. The open source engines do, and their source is the clearest description of what every hosted cache is doing underneath.

vLLM splits the KV cache into fixed blocks, 16 tokens each by default, and content-addresses them with a chained hash. Block 0's hash covers its own tokens; block 1's hash covers block 0's hash plus its own tokens, and so on down the prompt. The consequence is the whole rule in one line: if block N's hash matches, blocks 0 through N-1 are guaranteed identical, so the scheduler can find the longest cached prefix by walking hashes until one misses, without comparing a single token. That is also why nothing after a miss can ever be reused, no matter how much of it is unchanged text.

SGLang stores the same idea as a radix tree. Every completed request is inserted as a path, shared prefixes collapse into one copy of each unique node, and eviction runs LRU from the leaves inward, so a parent prefix survives until all of its children are gone. Cached blocks are page-aligned, which is the mechanical source of the minimum-prefix rule every vendor publishes: at a page size of 16, a 35-token sequence stores the first 32 tokens in two complete pages and leaves the last 3 uncached. A 300-token prompt has no full page worth keeping.

Why only prefixes, when the middle of a prompt repeats too

The obvious question, asked every time this topic reaches a forum: if a phrase recurs in the middle of many prompts, why can the engine not cache that too? The answer is that a token's cached vectors encode everything before it. As one commenter put it in the Hacker News thread on the subject, the model is effectively compressing the whole preceding context into each token, so the same words appearing after different context produce entirely different vectors. There is nothing to match on.

This is a research problem, not a law. Prompt Cache, published at MLSys 2024, makes non-prefix reuse work by declaring reusable segments as prompt modules and precomputing attention states that stay positionally valid wherever the module lands in a prompt, reporting time-to-first-token improvements from 8x on GPU inference to 60x on CPU inference. The reason no major provider sells it is that it requires you to author prompts against a schema. Every shipping product, including the paged, block-hashed caches above, descends instead from PagedAttention, which made cross-request KV reuse practical without changing how you write a prompt.

Why your cache is not shared with other customers

A tempting idea surfaces whenever people work this out: coding tools ship the same system prompt to every install, so a provider could cache it once globally and everyone would ride free. Providers segregate the cache per account instead, and the reason is a timing side channel. If a hit is measurably faster than a miss, and the cache spans tenants, then latency answers the question of whether anyone else sent this text recently. The Hacker News discussion works through the attack and its limits: guessing one token at a time would need the engine to store an entry after every single token, which no real implementation does because caches are block-aligned. The feasible version is coarser and still bad, confirming whether a competitor is loading a particular document or codebase into context.

The practical consequence is that a warm cache is something you create, not something you inherit. Cold start is real on every new conversation, and the only way to have a prefix already resident is to have sent it yourself, recently, to the same place.

Prompt Caching Pricing by Provider

Every number below was read from the vendor's own docs on September 1, 2026 (links in Sources). The cache read multiplier is cached input divided by base input. One representative model per vendor keeps the table readable; the same multipliers apply across each vendor's current lineup unless noted.

Two things stand out. First, the read multiplier has converged on 0.1x, with DeepSeek below it and Morph's DeepSeek V4 Flash and GLM-5.3 744B above it in ratio terms but far below it in absolute price: $0.03125 per 1M cached input on morph-dsv4flash is the lowest cached rate in the table after DeepSeek's own off-peak window, and Morph has no peak window (DeepSeek doubles every rate from 01:00 to 04:00 and 06:00 to 10:00 UTC on weekdays). Second, only Anthropic charges to write. A 1.25x write means a prefix must be read at least once more to break even against not caching; at 2x for the 1-hour TTL it must be read twice. Everyone else caches speculatively for free.

Per-vendor detail lives on the model pages: Anthropic API pricing, OpenAI API pricing, Gemini API pricing, DeepSeek API, and Kimi K3 API.

Anthropic Breakpoints: Four Slots and a 20-Block Window

Anthropic is the only major provider where you place the cache yourself, and the two limits that decide whether it works are easy to miss. There is a maximum of four cache_control breakpoints per request, and each breakpoint searches a bounded distance backwards. In Anthropic's wording: "The lookback window is 20 blocks. The system checks at most 20 positions per breakpoint, counting the breakpoint itself as the first. If the system finds no matching entry in that window, checking stops." A run of consecutive tool_use blocks counts as one position, and so does a run of consecutive tool_result blocks, so parallel tool calls do not single-handedly push the previous turn out of the window.

Two ways to place them. A single top-level cache_control field turns on automatic mode: the breakpoint lands on the last cacheable block and moves forward as the conversation grows, which is the right default for a chat loop. Per-block cache_control gives explicit control over exactly what gets written, which is what you want when the system prompt and the tool list should cache separately from the moving history.

Anthropic also publishes which changes invalidate which layer, and one row of that table matters more than the rest: changing tool definitions invalidates the tools cache, the system cache, and the messages cache, all three. Toggling web search or citations rewrites the system prompt, so tools survive and everything after does not. Changing tool_choice, adding or removing an image, or dropping a thinking block invalidates message blocks only. Read input_tokens carefully when you audit this: it counts only the tokens after the last breakpoint, not everything you sent, so the total is cache_read_input_tokens plus cache_creation_input_tokens plus input_tokens.

Worked Example: A 30-Turn Coding Agent

The session: a 20,000-token system prompt plus tool definitions, byte-identical every turn. Each turn appends 1,500 tokens of user message, tool results, and assistant reply, and the model writes 500 output tokens. 30 turns. Turn n re-sends the prefix plus n minus 1 turns of history, so prompt tokens across the session sum to 1,252,500. The "90% hits" column bills 1,127,250 of those at the cached rate and 125,250 at the input rate.

90% is conservative for this shape. With exact replay only the first prefill (20,000 tokens) and each later turn's new 1,500 tokens miss: 63,500 of 1,252,500, a 95% hit rate. Anything below 90% on a loop like this means the prefix is changing or the requests are landing on different workers. The usage object on the final turn of an exact-replay session looks like this:

{
  "usage": {
    "prompt_tokens": 63500,
    "completion_tokens": 500,
    "total_tokens": 64000,
    "prompt_tokens_details": { "cached_tokens": 62000 }
  }
}

62,000 of 63,500 prompt tokens bill at $0.03125 per 1M on morph-dsv4flash; the 1,500 new tokens bill at $0.1234375. Output is unaffected: caching never touches the decode side of the bill. If your agent's output share is large (always-on reasoning models like Kimi K3 emit long reasoning traces), caching cuts the input line and leaves the output line alone. For the full set of levers on the rest of the bill, see LLM cost optimization and Claude Code token usage.

Measured Hit Rates and Measured Savings

The arithmetic above assumes a hit rate. Here is what practitioners actually record. The most useful measurement is a controlled one: a bug report against qwen-code compared two clients driving the same Anthropic backend over comparable sessions, which isolates the client's prompt construction as the only variable.

The gap between 100% and 80% on the same model is entirely prompt construction. That is the honest ceiling to aim at: a coding agent that replays verbatim and lands on the same worker should read essentially all of its context from cache, and anything materially below that is a bug in the client, not a property of the cache.

On the savings side, two independent evaluations agree on the range. A PricewaterhouseCoopers team ran 500 agent sessions on DeepResearch Bench with 10,000-token system prompts across four models and measured cost reductions of 79.6% on GPT-5.2, 78.5% on Claude Sonnet 4.5, 45.9% on GPT-4o, and 41.4% on Gemini 2.5 Pro, with time to first token improving 13.0%, 22.9%, 30.9%, and 6.1% respectively. LangChain reported 49% to 80% token-cost reductions across its Deep Agents eval suite, including 77% on Claude Haiku and 49% on Gemini 3.5 Flash.

How It Breaks in Production

Every failure below is a filed, reproducible report against a shipping client, not a hypothetical. They are the five shapes this takes.

Every layer supports caching and the composition still does not

An engineer posted the bill from a normal local coding-agent workflow running through four layers, each of which supports prompt caching: an agent client, a proxy library, AWS Bedrock, and Claude Opus 4.6. Gross usage was $37,901.73, of which credits absorbed $8,026.54. The breakdown is the argument for this entire page.

Output was $698 of a $37,901 bill. Caching was not off, it was partial: 1.67B tokens did read from cache, against 6.47B that did not. Nothing in the chain reported a problem, because no layer is responsible for the composition. In the author's words, "prompt caching is supported" is not the same as "your actual agent stack is using prompt caching correctly", and budget alerts are not a kill switch. The only defence is to read the ratio of cached to uncached input tokens off your own requests on day one, before a month of them accumulates. Posted at Hacker News item 47933355.

All writes, no reads

The worst outcome is not a missing discount, it is paying the write surcharge on every single request. An OpenClaw bug report describes exactly that: caching configured, and every request showing a cache write of 170,602 tokens at $3 per 1M with cache reads pinned at zero. The cause was dynamic content in the system prompt, timestamps, message IDs, session metadata, and a "Current Date & Time" section, changing every turn. Measured cost went from an expected $0.05 per message to $0.50, and $35 in a day against an expected $9. On Anthropic this configuration is strictly worse than not caching at all, because the writes bill at 1.25x base input. Filed as openclaw issue 19534.

A side query that evicts the main thread

The qwen-code report contains a failure mode worth internalising, because a side query does not merely miss, it evicts. Its auxiliary calls resent the full conversation under a different system block and with no tools attached, so the prefix could never match the main agent's and returned cache_read = 0 by construction, in 22 of 129 requests. It then displaced the main conversation's cached prefix, so the next real turn missed too. One 45k-token uncached side query cost roughly three times the large cached main turn it interrupted. The fix is to run side queries behind the same system prompt and the same tool list as the main agent, appending the side instruction as a trailing message so it shares the prefix instead of competing with it. Filed as qwen-code issue 5942.

The agent's own tool call outlives the TTL

A 5-minute TTL is usually discussed as a problem for idle users. In an agent it is a problem for busy ones. A VS Code bug report puts it as a single rule: any two consecutive model calls with a time delta above 300 seconds miss the cache and pay full input price on the next call. The gap is not the human thinking, it is the agent's own npm install, test suite, or build finishing in a terminal. The report estimates a 3x to 5x cost multiplier for one mid-session gap and 8x to 15x cumulative on a long session with ten or more of them, and proposes a keepalive: a no-op request every 290 seconds while a tool runs, priced at a cached read plus one or two output tokens. Filed as vscode issue 321551. The alternative to a keepalive is a TTL that outlasts your longest tool call: Anthropic's 1-hour option at a 2x write, or on Morph a cache_ttl of 1h or longer at no write surcharge.

Your hit-rate metric is lying to you

Before concluding the cache is broken, confirm the number is real. Two open SGLang bugs describe a hit rate that reads wrong while the cache works fine: cached_tokens reporting zero when speculative decoding is enabled, and a cache_hit_rate gauge overwritten once per prefill batch, which makes it unusable under mixed traffic. Reported in sglang issue 20451 and issue 26608. Cross-check the billing line before you rewrite a prompt.

Cache-Aware Routing: Why a Naive Router Destroys the Hit Rate

An LLM router classifies a prompt and picks the cheapest model that can handle it. Applied per turn to a multi-turn session, it fights prompt caching in two ways.

• Each model has its own cache. A KV cache is a function of the weights. The prefix GLM-5.3 prefilled is useless to DeepSeek V4 Flash. The first turn on a new model prefills the whole conversation at that model's input rate.
• Within a model, the cache is per worker. A load balancer that spreads a conversation across replicas round-robin turns a 95% replay into a hit rate near 1 divided by the replica count. Session affinity, a stable mapping from conversation to worker, is what makes the automatic cache deliver.

Route once, at the session boundary

The first turn of a session is the one moment a switch is free, because nothing is cached yet. Morph's Model Router classifies difficulty, ambiguity, and domain in one call at $0.005 per request, returns the model to call from a candidate list you pin, and then gets out of the way. Every later turn reuses the decision and carries the same session key, so both the model and the worker stay fixed.

import os
import requests
from openai import OpenAI

MORPH_API_KEY = os.environ["MORPH_API_KEY"]
client = OpenAI(api_key=MORPH_API_KEY, base_url="https://api.morphllm.com/v1")

CANDIDATES = ["deepseek-v4-flash", "glm-5.2"]  # two tiers, not ten
_session_model: dict[str, str] = {}


def model_for_session(session_id: str, first_turn: str) -> str:
    # Route once, at the session boundary, when nothing is cached yet.
    if session_id in _session_model:
        return _session_model[session_id]
    resp = requests.post(
        "https://api.morphllm.com/v1/router/multimodel",
        headers={"Authorization": f"Bearer {MORPH_API_KEY}"},
        json={
            "input": first_turn,
            "allowed_models": CANDIDATES,
            "policy": "cost_efficient",
            "default_model": "glm-5.2",
        },
        timeout=5,
    )
    _session_model[session_id] = resp.json()["model"]
    return _session_model[session_id]


def handle_turn(session_id: str, messages: list[dict]):
    model = model_for_session(session_id, messages[-1]["content"])
    return client.chat.completions.create(
        model=model,
        messages=messages,
        extra_body={"prompt_cache_key": session_id},
    )

Two or three candidates is the right size. Every extra candidate is another cold cache the router can dump a session into. Re-classify only at boundaries you are already paying for: a new session, or a context compaction that rewrote the prefix anyway. Morph's Claude Code proxy ships this as a context lock: past 60k tokens of context it stops classifying and holds the session's model, because at that size prefill dominates any per-token rate difference.

Pin the worker with a session key

Caching on Morph needs no key. The key answers the other question: which worker serves the turn. Send one id per conversation as the prompt_cache_key body field or the x-session-id header, on every turn and on retries of the same turn, and each request routes to the worker that already holds its prefix. OpenCode and the Codex CLI already send prompt_cache_key; OpenRouter forwards x-session-id on behalf of its callers. The key changes placement, never billing. cache_ttl is independent: it controls how long that worker keeps the prefix, sliding, in five tiers from 5m to 24h. Invalid values are rejected with a 400.

from openai import OpenAI

client = OpenAI(
    api_key="YOUR_MORPH_API_KEY",
    base_url="https://api.morphllm.com/v1",
)

# One id per conversation, generated when it starts, reused on every
# turn and on retries of the same turn. It pins placement, not billing.
SESSION_ID = "conv-8f2c1a"

response = client.chat.completions.create(
    model="morph-dsv4flash",
    messages=messages,  # system, tools, full history, newest message last
    extra_body={
        "prompt_cache_key": SESSION_ID,
        "cache_ttl": "1h",  # sliding: every hit refreshes the clock
    },
)

usage = response.usage
cached = usage.prompt_tokens_details.cached_tokens
print(f"{cached}/{usage.prompt_tokens} prompt tokens served from cache")

curl -X POST "https://api.morphllm.com/v1/chat/completions" \
  -H "Authorization: Bearer YOUR_MORPH_API_KEY" \
  -H "x-session-id: conv-8f2c1a" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "morph-glm53-744b",
    "messages": [{"role": "user", "content": "..."}],
    "cache_ttl": "6h"
  }'

Escalate a turn without losing the cheap prefix

Some turns in a cheap session genuinely need the stronger model. The mistake is to move the session. Instead, run the hard turn on the strong model under its own session key, append its answer to the shared history, and continue the next turn on the cheap model. Appending never invalidates a prefix: the cheap model's cached blocks still match through the last turn it served, and only the new assistant message and the next user message prefill. The strong model pays one full prefill, which is the price of the escalation, not a recurring tax. If escalations become frequent, the classifier was wrong about the session; re-route at the next compaction.

CHEAP = "morph-dsv4flash"
STRONG = "morph-glm53-744b"


def handle_turn(session_id: str, messages: list[dict], hard: bool):
    # The cheap model keeps its warm prefix under the session key.
    # A hard turn runs on the strong model under its own key. Its answer
    # is appended to the shared history, so the cheap model's prefix is
    # still an exact match on the next turn: append never breaks a hit.
    model = STRONG if hard else CHEAP
    key = f"{session_id}:{model}"
    reply = client.chat.completions.create(
        model=model,
        messages=messages,
        extra_body={"prompt_cache_key": key, "cache_ttl": "1h"},
    )
    messages.append({"role": "assistant", "content": reply.choices[0].message.content})
    return reply

The same rule governs multi-agent layouts: give each subagent its own session key and its own stable prefix rather than sharing one conversation across models. See multi-agent model routing and, for Claude Code specifically, Claude Code Router.

Gateways and Sticky Routing

A gateway that proxies your call to one of several upstream providers adds a failure mode the vendor docs never mention, because a cache lives where it was written. If a later request routes to a different provider endpoint, that endpoint cannot read it, and nothing in the response tells you the discount vanished for a routing reason rather than a prompt reason.

OpenRouter documents how it handles this and the mechanism is worth understanding even if you never use it. It recognises a conversation by hashing the first system or developer message together with the first non-system message, then pins later requests to the endpoint that served the earlier ones. That heuristic breaks on exactly the agents that need it most: any agent that rewrites its opening messages changes the hash and loses the pin. Passing an explicit session_idreplaces the derived key, and the timing difference is the point. With a session id, stickiness starts after the first successful request. Without one, stickiness starts only after a cache hit has already been detected, so turn one is a coin flip. OpenRouter's own list of miss causes is short and matches everything above: the prompt is too short, the cache expired, the opening block keeps changing, or the request moved to a different provider.

The same class of bug appears when a gateway sits between an editor and a model. A VS Code report describes Claude models routed through OpenRouter with a bring-your-own-key setup getting no prompt caching benefit at all in agent mode, with the cost showing up as ordinary input tokens. Filed as vscode issue 312939.

Rules for a High Hit Rate

Measure, do not assume. Log cached_tokens divided by prompt_tokens per request and alert when a session's ratio drops below the replay expectation. On Anthropic the equivalent fields are cache_read_input_tokens and cache_creation_input_tokens; on DeepSeek, prompt_cache_hit_tokens and prompt_cache_miss_tokens. Claude Code exposes the same ratio in /cost.

Pitfalls

Kimi K3 needs reasoning_content replayed verbatim

Kimi K3 thinks on every turn and is trained to see its own prior reasoning. Moonshot's quickstart requires adding the complete assistant message to the next request, reasoning_content and tool_calls included, not content alone. Harnesses that strip the thinking to save tokens degrade the model and, because the stripped history differs from what the model saw, still miss the cache. Replay it and the prefix grows, which is exactly what $0.29 per 1M cached input on morph-kimik3 (against $2.50 uncached) is for. Details on the Kimi K3 API page.

Temperature does not break the cache

A misconception common enough that practitioners report arguing about it with people who should know better: sampling parameters do not affect cache hits. Temperature, top_p, and seed act during decode, choosing among the distribution the model produces. The cache is built during prefill, before any of them apply. Change the temperature between two otherwise identical requests and the second still hits. The parameters that do invalidate are the ones that change the serialized prompt, which on Anthropic means tool definitions, tool_choice, images, and thinking configuration. The test is simple: if it alters the bytes the model reads, it breaks the prefix; if it only alters how the next token is picked, it cannot.

cached_tokens is quantized, so it never equals your prompt length

A recurring confusion on OpenAI's developer forum: people compare cached_tokens against prompt_tokens, see a shortfall of a few hundred, and go looking for a prefix bug that is not there. Caching begins at 1,024 tokens and then advances in 128-token increments, so on a 6,260-token prompt the ceiling is 1,024 plus 40 times 128, which is 6,144 cached tokens. The remaining 116 are the unaligned tail, and they will always be there. The same forum thread is where the causal rule gets stated most plainly by practitioners: remove one word early in a prompt and everything after it is evicted, because attention at a position depends on every token before it.

Tool definitions that drift

Tool schemas usually sit right after the system prompt, so a change there invalidates nearly everything. Common causes: an MCP server that re-registers tools with a new description on reconnect, a framework that dedupes tools through a set and loses order, or a dynamic tool loader that injects a per-turn subset. If tools must vary per turn, place the variable subset after the stable ones.

Streaming retries

A stream that drops mid-generation is usually retried with the same request. That retry is a cache hit only if it reaches the same worker, so send the same session key on the retry. Do not append the partial assistant output to the history before retrying: the truncated message becomes part of the prefix on every later turn and differs from what any provider cached.

Editing history

Rewriting an earlier message, re-ranking retrieved chunks, or moving a summary to the top of the prompt changes the prefix at that point. Everything after it misses. This is expected; caching is prefix-based. Append, and when you must rewrite, do it once at a compaction boundary so the new prefix caches fresh and stays stable.

TTL expiry and eviction

Anthropic's default cache lives 5 minutes; a user who reads a long response and replies six minutes later pays a cache write again. OpenAI holds 30 minutes on GPT-5.6. Morph defaults to LRU with no fixed expiry and lets you set a sliding cache_ttl up to 24h for sessions that go idle between turns. Past the TTL the prefix stops hitting entirely, and re-sending it caches it again at the normal input rate; there is no write surcharge on Morph.

FAQ