A GPU's advertised hourly rate is half the story. Cost per million tokens, the number that actually sets your inference bill, is that rate divided by tokens per second, and tokens per second is the number almost nobody logs. Get that second number wrong by 10x, which is easy to do at low batch sizes, and your cost per million tokens is wrong by the same factor no matter how carefully you shopped for the hourly rate.
This post is the formula and the worksheet: the two inputs you actually need, how batch size and context length swing the result by 10 to 100x on identical hardware, and what those numbers look like right now across three GPU tiers. For the exhaustive model-by-model benchmark grid across Llama 4, Qwen 3, Gemma 4, DeepSeek V3, and Mistral, see our GPU cost per token benchmark. For the broader FinOps framework this formula feeds into, see AI inference cost economics.
The Two Numbers Behind Cost Per Million Tokens, and Why Most Teams Only Track One
Cost per million tokens has exactly two inputs: what you pay per GPU-hour, and how many tokens that GPU actually produces per second under your real traffic. Most teams track the first religiously, because it's printed on a pricing page and shows up on an invoice. They rarely track the second, because it means instrumenting the inference server, not reading a bill.
That gap is getting more expensive to ignore. Gartner forecasts that over half, 55%, of AI-optimized IaaS spending will support inference in 2026, rising to 59% in 2027, as more of the AI infrastructure budget shifts from one-time training runs to inference workloads that run continuously. Getting the per-token math right on the workload that's taking the larger share of the bill is the point of this post.
GPU $/hr Is the Number Everyone Has
Procurement negotiates it, dashboards display it, and every comparison post on the internet leads with it. On Spheron, on-demand rates are aggregated live across 5+ providers and the platform's certified data center network, with per-minute billing and no minimum rental period; on-demand rates currently start at $0.53/hr. That floor price moves with GPU model: an RTX 4090 sits at the $0.53/hr entry point today, while an L40S on-demand instance runs $0.96/hr. None of that tells you what a million tokens costs. It only tells you what an hour costs.
Pricing fluctuates based on GPU availability. The prices above are based on 03 Sep 2026 and may have changed. Check current GPU pricing → for live rates.
Tokens/sec Is the Number Almost Nobody Measures
Teams that skip this step usually fall back on one of two shortcuts: a vendor's single-request benchmark number, or nothing at all, judged instead by whether responses "feel fast." Neither one tells you what your production server does under your actual concurrency, your actual model, and your actual quantization. Continuous batching alone can move a GPU's real throughput by an order of magnitude versus a naive per-request loop; if you haven't measured tokens per second on your own stack this month, the number you're using to justify last quarter's GPU choice is probably stale.
The Formula: How to Calculate Cost Per Million Tokens
The formula is one line: cost per million tokens = cluster $/hr ÷ (tokens/sec × 3,600 ÷ 1,000,000). The hourly rate covers everything running for that deployment, tokens/sec is aggregate throughput measured on your own serving stack, and the 3,600 and 1,000,000 just convert seconds to hours and tokens to millions.
Worked Example at Two Different Throughput Levels
Here's the same 8x H100 SXM5 cluster, at the $2.90/hr per-GPU rate it was benchmarked at (8 × $2.90/hr for the full cluster), showing up with wildly different economics purely because of throughput. At batch size 1, the GPU spends most of its time waiting to load model weights from VRAM for a single active request; GPU utilization is under 5%, aggregate throughput is around 25 tokens/sec, and that works out to roughly $258 per million tokens. Push the same cluster to batch size 256 with continuous batching, and throughput rises to around 2,800 tokens/sec, dropping cost per million tokens to about $2.30.
That's a roughly 100x difference in cost per token, on hardware that didn't change and a price tag that didn't change. Batch size did all of it. If you want the mechanism behind why continuous batching closes that gap, our continuous batching explainer covers how iteration-level scheduling keeps every GPU slot filled instead of idling behind the slowest request in a batch.
Why Cluster Size, Not GPU Count, Is the Real Numerator
The numerator in that formula is the full running cluster's dollar-per-hour rate, not the sticker price of one GPU, and how many GPUs sit in that cluster is a deployment choice, not a fixed property of the model. A 70B model in FP16 needs roughly 140 GB of VRAM just to hold the weights, which fits on 2x H100 (160 GB combined) with headroom left for KV cache. The 8x H100 cluster in the worked example above isn't there for VRAM: our own 70B benchmarks run on 8x H100 because that configuration was chosen for maximum batch throughput, not the smallest footprint that gets the model loaded. Mix the two up, quoting the 2x-GPU rate against the 8x-GPU throughput or the other way around, and your cost per million tokens is wrong by up to 4x before you've measured anything else.
Batch Size and Context Length: How They Swing the Number
Two variables move cost per million tokens far more than any GPU model choice: how many requests you batch together, and how long the context those requests carry actually is. Both act on the same lever, GPU idle time, from different directions.
Batch Size 1 vs Continuous Batching, Same GPU
Static, single-request inference leaves a GPU idle between tokens and idle again waiting for the next request to arrive. Continuous batching, the scheduling approach vLLM runs by default and TGI and TensorRT-LLM implement under different names, refills a GPU's compute slot the instant a request finishes instead of waiting for a whole batch to complete together. That's the mechanism behind the roughly 100x swing in the worked example above, and it's why a serving stack running at effective batch sizes of 8 to 32 (a common outcome of under-configured max_num_batched_tokens settings) can sit 10 to 30x higher on cost per million tokens than one tuned for its real traffic.
What a Growing KV Cache Does to Decode Throughput as Context Grows
Longer context means a bigger cache, a bigger cache means less VRAM headroom for concurrent requests, and less headroom for concurrent requests means the batch size that made the mainstream-tier numbers above possible starts shrinking back toward the batch-1 case.
Our Fleet Numbers on Three GPU Tiers
Entry Tier, Mainstream Tier, Flagship Tier at Today's Live Rates
These three rows use measured throughput from our own benchmark posts paired with today's live Spheron rate for each GPU, so you can see the formula run on real numbers rather than a hypothetical.
| Tier | GPU | On-demand $/hr (live) | Workload | Measured throughput | Cost per million tokens |
|---|---|---|---|---|---|
| Entry | RTX 5090 | $0.86/hr | Llama 3.1 8B, FP16, batch 32 | ~3,500 tok/s | ~$0.068 |
| Mainstream | H100 SXM5 (1x) | $2.65/hr | Llama 4 Scout 17B MoE, vLLM | 4,200 tok/s | ~$0.18 |
| Flagship | B200 (8x, spot-only) | $3.90/hr per GPU ($31.20/hr cluster) | Llama 3.3 70B | 5,200 tok/s aggregate | ~$1.67 |
The entry- and flagship-tier CPM figures above are our own calculation: today's live $/hr from the pricing page run through the formula against the tok/s figures measured in our RTX 5090 benchmarks and GPU cost per token benchmark posts. The mainstream-tier row uses the same 4,200 tok/s that benchmark post measured, but at today's live $2.65/hr H100 rate rather than the $2.90/hr it was originally priced against, which is why the two posts show different numbers for the same throughput: $2.90/hr works out to roughly $0.19 per million tokens, $2.65/hr works out to roughly $0.18. That's the formula responding correctly to a rate that moved, not a rounding error. If your workload sits between entry and mainstream, our RTX PRO 6000 benchmarks run the same math at that workstation tier's real 30B AWQ and 70B FP8 throughput numbers.
Pricing fluctuates based on GPU availability. The prices above are based on 03 Sep 2026 and may have changed. Check current GPU pricing → for live rates.
Why the Flagship Row Can Flip Between On-Demand and Spot-Only Overnight
The flagship row above has no on-demand price because there isn't one right now: B200 is currently listed spot-only on Spheron, with spot capacity starting at $3.90/hr per GPU, cheaper but reclaimable at any time. That's not a fixed property of the GPU. Back when the cost-per-token benchmark post above was published, that same B200 SXM6 spot tier was priced at $2.07/hr per GPU, close to half of today's rate, which is why an 8x B200 spot cluster at $16.56/hr delivering 5,200 tok/s worked out to a $0.88 CPM then versus roughly $1.67 today at the same throughput. Whichever GPU sits at the top of your stack is the one most exposed to this kind of swing, because it's the tier with the least spare capacity sitting around. Model your flagship-tier number off the live pricing page the day you calculate it, not off a table from a few months back, and check whether an on-demand tier exists at all before you plan around one. If FP4 quantization is on the table for that tier, our FP4 on Blackwell breakdown and B300 vs B200 cost-per-token comparison both walk through how much further the number moves from there.
A Worksheet to Run This for Your Own Workload
Five Inputs to Pull From Your Own Logs Before You Calculate Anything
- Your actual billed $/hr, for the full cluster your deployment needs, not the per-GPU list price. Per-minute billing means this should already be close to your real invoice rate; confirm it against a recent bill rather than the pricing page's headline number.
- Aggregate tokens/sec at your production concurrency, pulled from your inference framework's own logs (vLLM, TGI, and TensorRT-LLM all expose this). A vendor's single-request benchmark is not this number.
- Your typical batch size or concurrent request count, at both P50 and peak. The worked example above shows why this single input can move your answer by two orders of magnitude.
- Average context length across your real traffic, input plus output. Longer contexts shrink the batch size your GPU can sustain, which quietly raises cost per million tokens even when nothing else changed.
- Your on-demand vs spot mix, if any traffic can tolerate interruption. Spot rates apply to a different reliability profile than on-demand, and the two shouldn't be averaged together into one blended number.
Common Mistakes That Quietly Inflate or Hide the Real Number
- Pricing a multi-GPU deployment off a single GPU's rate. If the model needs 8 GPUs to fit in VRAM, the numerator is 8 times the per-GPU price, every time.
- Reusing a vendor's batch-1 benchmark as your production throughput. It's usually the highest cost-per-token number that GPU will ever produce, not a representative one.
- Comparing a spot $/hr against on-demand throughput, or vice versa. Match the pricing tier to the reliability tier you're actually measuring.
- Not re-measuring after any model, quantization, or batching config change. A throughput number from before you switched to FP8, or before you tuned
max_num_batched_tokens, is not your current number. If throughput still isn't where you need it, speculative decoding is one of the few levers that raises it without a hardware change. - Ignoring idle GPU-hours. If a cluster runs at 40% utilization across the day, the honest cost per million tokens has to spread the idle hours across the tokens actually produced, not just the hours the GPU was busy.
Once you have your own number, it's worth checking it against two other reference points: the GPU cost optimization playbook for what to change first if the number is too high, and the LLM API pricing comparison if you want to see where self-hosting crosses over against paying a model provider per token instead. Teams optimizing for power draw rather than raw dollars can also run the same throughput numbers through a tokens-per-watt lens, and Spheron's docs cover how to get a deployment running to start pulling your own tokens/sec numbers in the first place.
Once you know your real cost per million tokens, comparing GPU tiers against live rates is the next step, and per-minute billing means the number you calculate is close to the number you'll actually pay.
Run the worksheet above against live rates before you commit to a tier.
Frequently Asked Questions
Two numbers, and most teams only have one of them logged: your cluster's real dollar-per-hour rate, covering every GPU the deployment needs rather than one GPU's list price, and aggregate tokens per second measured on your own serving stack at your production concurrency, not a vendor's single-request benchmark. Plug both into (cluster $/hr) / (tokens/sec x 3,600 / 1,000,000) and you get your real number. Skip the measurement step and you're pricing a GPU-hour, not a token.
Almost always it's batch size and context length, not the hardware. A GPU running at batch size 1 can show a cost per million tokens 50 to 100 times higher than the same GPU running continuous batching at high concurrency, because most of that GPU-hour is spent idle waiting on a single request instead of generating tokens for dozens of them at once.
No. A GPU that costs more per hour but produces proportionally more tokens per second can land at a lower cost per million tokens than a cheaper GPU with weaker throughput. The hourly rate only tells you the numerator; the denominator decides the answer, and you don't know it until you measure it.
Every generated token has to attend over the full KV cache, and that cache grows as context length grows, so decode throughput falls as conversations or documents get longer.
