The Economics of Prompt Caching
Anthropic’s prompt caching beta changes the economics of long prompts: writing to the cache costs 25 percent more than regular input tokens, while reading costs 90 percent less. We work through the prices and break-even point, examine the five-minute TTL, and derive the implications for RAG pipelines and agent loops.
The Cost of Repeating Yourself
Large language model APIs are stateless. Every request therefore carries the full context again: the system prompt, tool definitions, few-shot examples, and reference documents. The provider bills every token at the full input rate on every call — even when 95 percent of the prompt matches the previous request. With long, mostly static prompts, that repetition quickly becomes the largest cost component.
The numbers are concrete. A 100,000-token context on Claude 3.5 Sonnet costs $0.30 per request at the base input price of $3 per million tokens. At 1,000 requests per day, that is $300 per day for tokens the model has processed hundreds of times before. Until now there was no way to tell the API: this part is already known.
What Anthropic Shipped
In August 2024, Anthropic released prompt caching as a public beta on the Anthropic API, initially for Claude 3.5 Sonnet and Claude 3 Haiku, with support for Claude 3 Opus announced to follow. The feature is enabled per request via the HTTP header anthropic-beta: prompt-caching-2024-07-31 and cache_control markers on individual content blocks.
The mechanics are strictly prefix-based. The cacheable prefix spans tools, system prompt, and messages, in exactly that order, up to a cache breakpoint; up to four breakpoints are allowed per request. Cached segments live for five minutes, and every cache hit resets that timer. Minimum cacheable length is 1,024 tokens on Claude 3.5 Sonnet and 2,048 tokens on Claude 3 Haiku.
Write and Read Pricing
Prompt caching introduces two new token prices. Writing a prefix into the cache costs 25 percent more than the base input rate. Reading it back costs 10 percent of the base input rate — a 90 percent discount. Output tokens are unaffected. Anthropic states cost reductions of up to 90 percent and latency reductions of up to 85 percent for long prompts.
The pricing structure is deliberate: the 25 percent premium covers storing the prefix, while the 90 percent discount rewards its reuse. There is no separate storage fee and no capacity charge — access frequency alone determines whether caching pays. That makes the model easy to reason about, but just as easy to misuse on low-traffic endpoints.
| Model | Base input ($/MTok) | Cache write ($/MTok) | Cache read ($/MTok) |
|---|---|---|---|
| Claude 3.5 Sonnet | 3.00 | 3.75 | 0.30 |
| Claude 3 Haiku | 0.25 | 0.30 | 0.03 |
| Claude 3 Opus (announced) | 15.00 | 18.75 | 1.50 |
The Break-Even Arithmetic
The arithmetic is simple. A prefix used N times inside the TTL window costs 1.25 + (N − 1) × 0.1 in units of the uncached price, instead of N. At N = 2 that is 1.35 versus 2.0 — the second call already amortizes the write premium and saves 32 percent. At N = 10 the saving reaches 78 percent, converging toward 90 percent.
Conversely, caching is not an automatic discount. If an application reuses a prefix less than once every five minutes, it pays the 25 percent write premium on every call without receiving a discounted read. Simon Willison put it plainly on release day: apps prompting less than once every five minutes lose money. Prompt caching is therefore a deliberate bet on request frequency.
What Prompt Caching Does Not Do
Prompt caching is exact-prefix matching, not semantic caching. A single changed byte early in the prompt — a timestamp, a user ID, a reordered tool definition — invalidates everything after it. Volatile content therefore belongs at the end of the prompt, never at the beginning. Nothing about the model's context window changes; cached tokens still count toward it in full.
It is also not a persistence layer. The five-minute TTL makes the cache an optimization for bursts of activity, not a stored knowledge base. It saves nothing on output tokens, which remain the most expensive part of every response. Prompts below the minimum thresholds are not cached at all. And the feature is a beta: pricing and semantics may still change.
Comparison with Gemini Context Caching
Google shipped context caching for Gemini 1.5 Pro and 1.5 Flash in June 2024 with a different model: a 75 percent discount on cached input tokens plus a storage fee of $4.50 per million tokens per hour for 1.5 Pro ($1 for Flash), a minimum of 32,768 tokens, and explicitly managed cache objects with configurable TTL.
The designs target different workloads. Gemini's hourly storage fee favors a few very large, long-lived caches that are queried continuously. Anthropic's scheme has no storage fee, a low 1,024-token minimum, and a short self-refreshing TTL — it favors high-frequency access patterns such as chat sessions and agent loops. Neither is strictly cheaper; the workload decides.
Implications for RAG and Agents
For RAG systems the pricing shifts a boundary. Reading 200,000 cached tokens on Claude 3.5 Sonnet costs $0.06 per request. For small and mid-sized corpora that fit into the context window, placing the entire document set into a cached prefix now competes with chunk retrieval — no retriever, no index, no chunking artifacts. Retrieval remains necessary for large or fast-changing corpora, and retrieved chunks belong after the last cache breakpoint.
For agents, the leverage is greater still. An agent loop resends its growing conversation history with every tool call; those calls typically arrive seconds apart, comfortably inside the five-minute TTL. Caching the history prefix cuts each iteration's input cost by up to 90 percent. Long tool definitions and many-shot examples also shift from a per-call tax to near-free reads after the first write. At Blue IT Systems, we consider this the strongest case for the feature.
Outlook: Caching as Default Infrastructure
Our expectations, written in August 2024: caching will become default infrastructure rather than a beta flag. Competing APIs will converge on some form of it, and TTL options will grow beyond five minutes where workloads demand it. Billing models will differentiate further — write premiums, storage fees, or both.
Prompt design will change with the economics. We expect cache-first prompts: a stable prefix of instructions, tools, and examples, followed by all volatile content. That discipline weakens one argument for fine-tuning and strengthens long-context architectures relative to retrieval for mid-sized corpora. Individual predictions age quickly in this field; the economic direction is more durable: pay once for static context rather than on every call.