Token Cost Scaling Context Window: The 128K Bend (2026)

Token cost scaling context window length is the real shape of the problem: cost per token is not a constant you multiply by volume, it's a function of how much context sits in front of each token, and that function bends. Prefill compute grows quadratically with attention math, and a growing KV cache eats the concurrency a GPU needs to keep per-token cost down in the first place. Put the two together and the real $/1M-token curve turns upward well before 128K, and you can already see the bend in what Gemini charges today and what pre-4.6 Claude models used to charge, even on pricing pages that otherwise look flat.

TL;DR: Token Cost Scaling Context Window and the 128K Bend

• Prefill: the attention term in a transformer's forward pass grows with context length; for a Llama 3.1 70B-class model, it equals the parameter term at about 107,000 tokens.
• KV cache: at FP8, that model's cache costs about 160KB per token, so cache size doubles every time context length doubles, which is what drives the concurrency drop below.
• Concurrency: that VRAM math limits one GPU to roughly 24 concurrent 16K sessions but only about 3 at 128K.
• Pricing tiers: Gemini 3.1 Pro's input rate doubles, from $2.00 to $4.00 per million tokens, past 200K tokens.
• Spheron: bills by the GPU-hour, so $5.26/hr/hr on Spheron's H200 is the same rate at 4K or 128K tokens.

Why Token Cost Scaling Context Window Isn't Constant

A transformer's forward pass has two costs baked into every token: running the weights, and attending to everything already in context. The first is fixed per model. The second scales with how much context exists, and it scales in two different ways depending on whether the GPU is filling the context (prefill) or generating into it (decode).

This site's own cost-per-token formula post already covers the concurrency side of this at the formula level: how batch size, quantization, and context length each move the tokens-per-second denominator. This post goes underneath that formula, into the FLOP and byte math that explains why context length also reshapes prefill cost, not just decode concurrency, and pins down exactly where the bend starts for a real model shape.

Prefill Is Quadratic: The Attention Term Vendors Drop From Their FLOP Math

Adam Casson's widely cited FLOP-cost approximation for a transformer forward pass writes the per-token compute as C_forward(n_ctx) ≈ 2N + 2·n_layer·n_ctx·d_model, where the first term is the parameter-driven cost that barely moves with context length, and the second is the attention term that grows directly with it. Sum that per-token cost across every position in a prompt of length n_ctx and the attention term's contribution grows with the square of the sequence length, while the parameter term only grows linearly. That's the textbook shape of "prefill is quadratic," and it's a property of how many tokens attend to how many other tokens, not an implementation detail any one vendor chose.

It's also not something FlashAttention fixes. FlashAttention 2 and 3 rewrite how attention reads and writes memory during the computation, which is why they cut wall-clock time and unlock FP8 attention on Hopper hardware, but neither changes the number of floating-point operations attention has to perform. The FLOP count stays quadratic; FlashAttention just gets you to the answer faster per FLOP.

Decode Is Memory-Bandwidth Bound: KV Cache Eats the Concurrency That Made Tokens Cheap

Prefill is compute-bound and quadratic. Decode, generating one token at a time after the prompt is processed, is a different problem: it's memory-bandwidth bound, because each step has to read the model's weights and the full KV cache for every active sequence out of HBM to produce a single new token. kipply's transformer inference arithmetic gives the per-token KV cache cost directly: KV cache memory per token is 2 x 2 x n_layers x n_heads(kv) x head_dim bytes at 16-bit precision, where the two factors of 2 are for storing both K and V, and for bytes per element.

That formula is why long context is expensive on the decode side even when nothing else changes. A longer conversation means a bigger KV cache per sequence, which means fewer sequences fit in the same VRAM budget, which means a GPU that could batch 24 short conversations together can only batch a handful of long ones. Fewer sequences sharing a decode step is exactly the concurrency this site's guide to context engineering and KV cache economics covers from the serving-stack side; this post is the FLOP-and-byte math underneath it.

Where the Bend Actually Starts: The Crossover Point for Real Model Shapes

Casson's formula gives an exact answer for where the attention term catches up to the parameter term: it happens at n_ctx = N / (n_layer × d_model). Plug in a real 70B-class shape and the number is concrete, not abstract.

Llama 3.1 70B's published architecture is hidden_size 8192, 80 transformer layers, 64 query heads and 8 KV heads. Using its roughly 70-billion-parameter count as N, n_layer of 80, and d_model of 8192, the crossover works out to 70,000,000,000 / (80 × 8,192), or about 106,800 tokens, roughly 107,000. Below that point, running the weights dominates the per-token cost. Above it, attending to context does. That crossover sits well inside a 128K context window, which is exactly why 128K is the point where a lot of production serving stacks start feeling the bend rather than the point where it begins.

Push past a few hundred thousand tokens and the quadratic term stops being a rounding error and starts being the whole story, which is the problem ring attention and sequence parallelism exist to solve by spreading that attention compute across multiple GPUs instead of asking one GPU to eat the full n² term alone.

The Cost Curve: Relative $/1M Tokens at 16K, 32K, 64K, and 128K

Two separate effects compound as context grows: prefill gets more expensive per token, and decode concurrency drops. Here's what each looks like worked through with real numbers instead of left as an abstraction.

The Worked Example: Llama 3.1 70B FP8 KV Cache vs Fixed VRAM Budget

Start with the prefill side. Using Casson's formula at four common context checkpoints (16,384 / 32,768 / 65,536 / 131,072 tokens, chosen as clean powers of two rather than the marketing-friendly "16K/32K/64K/128K" labels), the attention term's share of per-token forward-pass compute climbs steadily:

By 64K, prefill already costs 61% more per token than it does at negligible context. By 128K, it's more than double.

Now the decode side, which is where KV cache VRAM does its damage. At FP8, kipply's formula puts Llama 3.1 70B's KV cache at about 160KB per token (80 layers × 8 KV heads × 128 head dimension × 2 for K/V × 1 byte per element at FP8, roughly half the 16-bit figure the formula gives directly). A 70B model's weights at FP8 occupy roughly 70GB on their own, which leaves an H200 141GB with about 71GB free before reserving anything for activations, the serving framework's own overhead, and PagedAttention block fragmentation. Reserve a conservative 10GB for that overhead and roughly 60GB is left for KV cache. An H100's 80GB doesn't leave nearly enough room for that same model plus any meaningful cache, which is the practical reason a 70B-class, long-context deployment reaches for the larger card in the first place.

Concurrency Loss Table: 16K to 128K on a Single GPU

Divide that ~60GB KV cache budget by the per-sequence cache size at each context length and the concurrency ceiling on a single GPU falls in a clean, predictable pattern, because KV cache size doubles every time context length doubles:

That's an 8x drop in concurrency from 16K to 128K, driven purely by VRAM, before accounting for the fact that each decode step at 128K also has to read a KV cache eight times larger than the 16K case to produce that same next token.

From Concurrency to Cost Per Million Tokens

Decode throughput on a memory-bandwidth-bound GPU scales roughly with how many sequences are batched together, up to the point where the GPU runs out of either compute or VRAM. Holding per-sequence decode speed roughly constant, an 8x drop in concurrency translates to a roughly 8x drop in aggregate tokens generated per GPU-hour, which is the same as saying cost per output token is roughly 8x higher at 128K than at 16K from concurrency loss alone.

Stack that against the 2.23x prefill inflation from the table above and a request that mixes a long prompt with a long generation can plausibly cost several times more per token at 128K than the same workload at 16K, well before any vendor adds margin for the context tier itself. This is a capacity-planning model built from published VRAM specs and formulas, not a benchmarked throughput number from a specific serving stack; the exact multiplier will move with your batch scheduler, your quantization scheme, and how much headroom you actually reserve for activations. The direction and the rough shape of the bend won't.

What This Means for Pricing Long-Context Features to Customers

If the underlying compute cost bends upward with context, a vendor's public pricing has three honest options: charge more for long context, eat the difference, or engineer it away. What the frontier labs actually do splits across all three.

What the Frontier Labs Already Charge for Long Context (Gemini vs Claude)

Google prices long context as a distinct, published tier. Gemini 3.1 Pro's input rate doubles from $2.00 to $4.00 per million tokens once a request crosses 200,000 tokens, and its output rate rises 1.5x, from $12.00 to $18.00 per million tokens. Gemini 2.5 Pro uses the identical 200K breakpoint and the identical 2x/1.5x multipliers, just off a lower base: $1.25 to $2.50 per million input tokens, $10.00 to $15.00 per million output tokens.

Anthropic has moved the other direction. Its published pricing states that Claude 4.6 and later models, including Claude Mythos Preview, bill the entire 1M-token context window at standard per-token rates, so a 900K-token request costs the same per token as a 9K-token one. That's a reversal from Anthropic's own earlier models, which carried a long-context premium the way Gemini still does. GPT-5.6 Sol's pricing is a third recent data point worth reading alongside these two if you're weighing a managed API against self-hosting for a specific workload.

Google and Anthropic's rates above are their published list prices as of the sources cited; check each vendor's own pricing page before budgeting against them, since API pricing changes independently of anything on this page.

A Flat Long-Context Price Is Either a Subsidy or an Efficiency Win, Not a Law of Physics

The math in the sections above doesn't go away just because a price sheet is flat. If a lab removes its long-context premium while the underlying prefill-and-KV-cache cost curve still bends, one of two things is true: the company is eating a thinner margin on long-context requests to win developer adoption, or it has engineered its own serving stack (disaggregated prefill and decode, more aggressive prefix caching, better attention kernels) hard enough that its internal cost curve is flatter than the generic one this post derives. Either is a legitimate business decision. Neither one repeals the compute cost itself; it just decides who absorbs it. The broader GPU FinOps playbook for reasoning about that trade-off across a whole inference budget, not just the context-length slice of it, is worth reading if you're setting this policy for your own product rather than just consuming someone else's.

How to Price Your Own Long-Context Tier if You're Serving on Rented GPUs

If you're running your own model on rented hardware rather than calling a managed API, the concurrency table above is the calculation to run against your own model shape and your own GPU before you set a price:

1. Compute your model's KV cache bytes per token using kipply's formula for your architecture, then size it at every context length you plan to support.
2. Subtract weights plus a real overhead reserve from your GPU's VRAM to get an honest KV cache budget, not the full nameplate VRAM figure.
3. Divide that budget by KV cache size per sequence at each context length to get your real concurrency ceiling, the way the table above does for a 70B model on an H200.
4. Convert concurrency to GPU-hours per million output tokens using your own measured decode throughput, not an assumed one; the shape of the curve is transferable, the absolute number is not.
5. Decide whether a flat price recoups margin at your longest supported context, or whether you need a Gemini-style breakpoint, given how much concurrency you actually lose there.

Spheron's own pricing is a useful example of where the infrastructure layer stops caring about any of this: GPU rentals bill per GPU-hour with per-minute billing after a 20-minute minimum, regardless of what context length or model you run on the hardware, so $5.26/hr/hr for an on-demand H200 (or $2.74/hr/hr on spot, for latency-tolerant batch jobs where reclaim without notice is an acceptable trade) is the same number whether your average request is 4K tokens or 128K. The concurrency math above is still yours to do; renting the GPU doesn't change how much KV cache a 128K context needs, it just decides what you pay for the hardware that holds it. It also doesn't hand you a serving stack: you still have to run and benchmark vLLM or SGLang yourself to know your real tokens-per-second at each context length, and if your call volume is too low to keep a rented GPU busy, a managed API's long-context premium is still probably the cheaper option in practice.

Pricing fluctuates based on GPU availability. Spheron rates above are live as of 20 Sep 2026; other providers reflect their most recent published rates and may have changed. Check current GPU pricing → for live rates.

The bend in the curve is structural to how attention and KV cache work, not a pricing choice any single vendor made. Whether you're evaluating a managed API's long-context tier or setting your own, the crossover point and the concurrency table above are the two numbers that actually explain why the sticker price stops looking flat once a prompt gets long.

A flat per-GPU-hour rate doesn't erase the prefill and KV cache math above, but it does mean your infrastructure cost stops being a function of context length the moment you stop renting by the token.

Get started on Spheron →