What Is Prefix Caching? Why Repeated Prompts Get Cheaper

Prefix caching skips redundant GPU work by storing the KV cache tensors from a previous request's tokens and reusing them for any new request that starts with the same tokens. It does not make the model faster, and it does not touch every part of an inference request. It removes the prefill computation for whatever prefix has already been seen, which is exactly the reason agent and RAG workloads see the biggest gains and one-shot creative prompts see almost none.

TL;DR: What Is Prefix Caching?

• What it caches: KV cache tensors from prefill, keyed to the exact token sequence, not model weights or output text.
• What it skips: prefill for sequences already seen. Decode never changes, since it always makes one new token at a time.
• Two engines: vLLM hashes 16-token blocks; SGLang's RadixAttention uses a radix tree, hitting up to 6.4x higher throughput on prefix-heavy traffic.
• Not prompt caching: OpenAI, Anthropic, Gemini, and DeepSeek's "prompt caching" is a billing wrapper around this same mechanism.
• Hit rate: Character.AI runs KV cache reuse at 95% while serving over 20,000 queries per second.
• Fleet sizing: Spheron bills bare-metal H100s at $2.65/hr/hr on-demand; size the KV cache pool a hit rate needs against how much GPU memory your model actually needs.

What Is Prefix Caching? What It Actually Caches (and What It Doesn't)

The name is precise: it caches a prefix, meaning a contiguous run of tokens starting from position zero, and it caches the KV tensors that prefix produced, not anything about the request's meaning or its eventual output. Two requests that both start with the same 800-token system prompt and tool schema get to share that portion of the KV cache; two requests that mean the same thing in different words share nothing, because the cache matches on tokens, not semantics.

KV Cache Recap: Why Prefill Is the Expensive Part It Skips

Every autoregressive decode step attends over the key and value tensors of every token that came before it. Computing those K and V tensors for a token is what happens during prefill, and it's the part of a request that scales with the full length of the input: a 4,000-token prompt means computing K and V for all 4,000 tokens before the model can generate its first output token. That's also why KV cache size itself is a live capacity question. Architectures with fewer KV heads shrink the tensors that get stored and reused; see our breakdown of what grouped-query attention actually saves in KV cache memory for the mechanics behind why a 70B model on GQA needs a fraction of the cache a comparable MHA model needs, independent of whether that cache gets reused across requests.

Once prefill finishes, decode takes over, and decode generates exactly one new token per forward pass, attending back over whatever KV cache exists so far, cached or freshly computed. There's no way to skip decode. Every output token is new by definition. This is the load-bearing fact behind everything else in this post: prefix caching is a prefill optimization only, and its ceiling is however large a share of your total compute prefill represents.

Block Hashing (vLLM) vs Radix Tree (SGLang): Two Implementations, One Idea

As the vLLM design documentation puts it: "we hash each kv-cache block by the tokens in the block and the tokens in the prefix before the block." A block has to be completely full before it becomes eligible for caching, so a shared prefix shorter than 16 tokens gets no benefit at all. When the cache runs out of room, vLLM evicts using LRU, dropping the least-recently-used head block from its free queue and removing its hash table entry.

SGLang takes a different data structure to the same problem. RadixAttention stores cached KV tensors in a radix tree keyed by token sequence: a new request walks the tree from the root, matching as far as it can, and only computes fresh nodes for the point where its tokens diverge from anything already cached. It uses LRU eviction the same way vLLM does, but the tree structure lets it share partial-prefix matches across many different downstream branches more naturally than fixed block boundaries do, which matters for workloads like few-shot prompting or tree-structured agent search where many requests share a common root but fan out differently after that. We ran both approaches head-to-head with real throughput numbers in vLLM vs SGLang: RadixAttention vs PagedAttention benchmarks if you want hardware-level results rather than the mechanism on its own. Both engines build their cache reuse on top of the same block-level memory management that underlies continuous batching more broadly.

What It Doesn't Save: Decode Tokens, Unique Prefixes, Reordered Context

Three things break or limit prefix caching, and all three come directly from the fact that it matches on exact leading tokens:

• Decode tokens. Already covered above, but worth stating plainly: no cache scheme changes decode cost, because decode has nothing to cache. Speculative decoding attacks that half of the request instead; see speculative decoding explained for the decode-bound counterpart to this prefill-bound optimization.
• Genuinely unique prefixes. If every request opens with a personalized system prompt (a user's name, their account state, a timestamp) baked into the first tokens, there's no shared prefix to hash, and the hit rate sits near zero no matter how the cache is tuned.
• Reordered context. Because the cache keys on exact token sequence, a RAG pipeline that retrieves the same three document chunks in a different order on the next call produces a different token sequence from position one, and the match breaks immediately even though the content is identical. Prefix caching cares about token order, not information content, which is the exact distinction that separates it from semantic caching. Our semantic caching for LLM inference guide covers the layer that matches on meaning instead of tokens, useful context for the next section.

Prefix Caching vs Prompt Caching: Don't Confuse the Two Layers

These two terms get used interchangeably in vendor docs and it causes real confusion. They are not the same layer of the stack, and knowing which one you're talking about changes who controls the discount.

API-Level Prompt Caching (OpenAI, Anthropic, Gemini, DeepSeek)

Prompt caching, as OpenAI, Anthropic, Gemini, and DeepSeek use the term, is a billing feature exposed through a hosted API. You send tokens, the provider's infrastructure decides whether it recognizes a matching prefix from a recent call, and if it does, you get charged less for those tokens on the second call. OpenAI's Prompt Caching activates automatically once a prompt exceeds 1,024 tokens on GPT-5.6 and later, where a cached prefix also stays eligible for reuse for 30 minutes after its last write or reuse and the API reports the exact eligible boundary with no rounding; earlier models instead match the longest previously-seen prefix in 128-token increments. Anthropic's prompt caching works differently on the economics: it charges a 25% premium on the initial cache write and gives a 90% discount (10% of the base token price) on every subsequent cache read. DeepSeek's version is disk-backed rather than memory-only; its on-disk context caching cuts the price of a cache-hit token by up to 90%, roughly a tenth of the cache-miss price in its listed example, depending on the model tier.

Every one of those is a pricing decision made on the provider's own infrastructure. You never see the cache, you can't inspect its hit rate directly beyond whatever usage metadata the API returns, and you can't tune block size or eviction policy. That's the trade-off of API-level caching: someone else decides what gets discounted and by how much.

Server-Level Prefix Caching (vLLM, SGLang): Where the Discount Actually Comes From

Every one of those API-level features is a thin billing layer sitting on top of the exact mechanism described in the section above: a server holding onto KV cache tensors and matching new requests against them. When you self-host with vLLM or SGLang, there's no separate "prompt caching" product to enable and no premium-then-discount pricing schedule, because you're not paying per token to a third party. You get the underlying mechanism directly: turn on Automatic Prefix Caching or RadixAttention (SGLang runs it on by default), and every matching prefix skips its prefill compute on your own GPU. The "discount" is simply the GPU time you don't spend, which shows up as more throughput per hour rented rather than as a line item on an invoice. This is also why prefix cache hit rate matters more when you self-host: it isn't a number a vendor reports back to you after the fact, it's a property of your own traffic and your own cache configuration that you can measure and tune directly.

Prefix Cache Hit Rate Economics: When Reuse Pays Off vs When It's Dead Weight

A hit rate on its own tells you almost nothing about cost savings. A 90% hit rate on a workload where prefill is 5% of total compute barely moves your GPU bill; a 40% hit rate on a workload where prefill is 70% of compute moves it substantially. The number that actually predicts savings is the product of the two.

The Formula: Hit Rate x Prefill Fraction = Cost Saved

Prefix cache hit rate is the share of prefill tokens that arrive already cached instead of needing fresh computation. Prefill fraction is the share of a request's total GPU compute that prefill represents in the first place, which depends on your input-to-output token ratio: long prompts with short generations are prefill-heavy, and short prompts with long generations are decode-heavy. Multiply the two and you get the fraction of total per-request compute that prefix caching actually removes:

The rows that lose the least compute aren't the rows with a low hit rate on their own; they're the rows where hit rate and prefill fraction are both modest, or where one of the two is near zero. A workload can have a respectable hit rate and still see almost no cost benefit if prefill was never a large share of its compute to begin with.

Real-World Hit Rates: Character.AI's 95% and What Drops a Hit Rate to Zero

Character.AI's engineering team describes indexing cached KV values by a rolling hash of prefix tokens in a tree-structured LRU cache, and routing repeat queries from the same dialogue to the same server through sticky sessions, which is what makes a 95% cache hit rate achievable while serving over 20,000 inference queries per second. Their own description of the payoff is direct: "each server can cache thousands of dialogues concurrently." That number is high because chat is close to the best case for prefix reuse: the same conversation history gets resent turn after turn, and sticky routing guarantees the request lands on a server that already holds the matching cache rather than a cold one.

The number drops toward zero the moment any of the three failure modes from the earlier section shows up at scale: personalized prefixes with no shared structure, retrieval pipelines that reorder chunks between calls, or load balancing that routes requests round-robin instead of by cache affinity. That last one is an infrastructure problem, not a caching problem, and it's solvable: a request-router that's aware of which server holds which cache preserves a hit rate that plain round-robin balancing would otherwise destroy.

How This Changes Your GPU Count for Agent and RAG Workloads

Why Agent and RAG Traffic Are the Best Case for Reuse

Agent and RAG traffic tend to combine both halves of the hit-rate formula in your favor at once. An agent loop resends its full system prompt, tool definitions, and prior reasoning steps on every call in the loop, so the prefix grows and repeats within a single task; a RAG pipeline resends the same retrieval-template scaffolding and, if chunk ordering is kept stable, the same retrieved passages across many user queries hitting the same document set. Both patterns push hit rate up and, because those shared prefixes tend to be long relative to a short generated answer, push prefill fraction up too. The SGLang paper reports its throughput gains specifically "on tasks including agent control, logical reasoning, few-shot learning benchmarks, JSON decoding, retrieval-augmented generation pipelines, and multi-turn chat" for exactly this reason, measuring up to 6.4x higher throughput than prior state-of-the-art inference systems on that mix. For the fuller agent-economics picture beyond prefix caching alone, including how context length compounds with cache hit rate over a long agent session, see context engineering for production AI agents.

Worked Example: Sizing a Fleet With and Without Prefix Caching

Take a RAG support bot with a 3,000-token prompt (a 2,400-token system prompt plus retrieved document template that stays fixed across most calls, plus 600 tokens of unique user query and top retrieved passage) and a 250-token generated answer. Assume prefill is roughly proportional to input tokens processed and this traffic pattern sees an 85% hit rate, meaning on average 2,550 of the 3,000 input tokens arrive already cached and only 450 need fresh prefill.

Without prefix caching, every one of those 3,000 tokens runs through prefill on every call. With it enabled, the same request only prefills the 450 uncached tokens, roughly an 85% cut in prefill compute for that request type. If prefill was consuming somewhere around 60% of this workload's total per-request GPU time before caching (a reasonable split for a prompt this size against a short answer), that's a ~51% reduction in total compute per request once the cache is warm. In practice that headroom doesn't disappear, it gets converted into either serving roughly twice the request volume on the same GPU count, or running the same volume on meaningfully fewer GPUs, since the freed prefill slots let the scheduler pack more concurrent decode work into every batch. The exact ratio depends on your batch scheduler and how continuous batching fills the freed capacity, since iteration-level scheduling is what turns freed prefill slots into more concurrent sequences rather than idle GPU time.

This is also a workload worth measuring on your own traffic rather than trusting a general formula, since hit rate and prefill fraction both vary by prompt template and retrieval design. Spheron's bare-metal H100 instances (currently $2.65/hr/hr on-demand or $2.10/hr/hr spot) give root-level access to set --enable-prefix-caching on vLLM or run SGLang with RadixAttention on directly, so you can log your own hit rate against real traffic on billed-per-minute infrastructure before committing to a fleet size built around an assumed number. That's a materially different starting point than a managed inference endpoint, where the caching layer, if one exists at all, isn't something you can inspect or tune.

Pricing fluctuates based on GPU availability. Spheron rates above are live as of 18 Sep 2026; other providers reflect their most recent published rates and may have changed. Check current GPU pricing → for live rates.

Where Prefix Caching Won't Shrink Your GPU Count

Three traffic shapes see little or no benefit, and it's worth ruling them out before assuming a caching rollout will cut your fleet:

• Decode-bound workloads. Long-form generation with short, unique prompts (summarization of a long document into a long summary, extended creative writing) spends most of its compute in decode, which caching can't touch regardless of hit rate.
• Genuinely one-shot traffic. If no two requests share a meaningful prefix, whether because prompts are fully personalized or because the product simply doesn't have repeat structure, there's nothing for the cache to match against.
• Cache thrashing under memory pressure. If GPU memory is tight enough that the LRU eviction policy is constantly discarding blocks before they get reused, you pay the bookkeeping cost of hashing and lookups without collecting the benefit. This is a capacity problem before it's a caching problem, and it's the same VRAM math that governs KV cache sizing generally.

None of these are reasons to skip prefix caching. vLLM and SGLang both make it close to free to enable, and it will never make decode-bound or one-off traffic slower. It's a reason to measure your own hit rate and prefill fraction before promising a specific GPU count reduction to whoever is budgeting the fleet.

Sizing a GPU fleet around a real prefix cache hit rate takes access to raw engine flags, not a managed endpoint that hides the KV cache from you.

Rent H100 GPU on Spheron →