Every LLM API cost comparison on the internet reduces to the same two numbers: what the GPU costs per hour, and how many tokens it produces per second. Get the second number wrong and the comparison is meaningless no matter how careful the first one is. Most published tokens-per-second figures come from a vendor's peak-batch benchmark, not from a server actually answering user requests, so the cost-per-token math built on them is wrong before you've divided anything. This post is the formula itself: how to go from a GPU's hourly rate to a defensible dollar-per-million-token figure, why your measured number will differ from the vendor's, and exactly how much batch size, quantization, and context length move that number.
TL;DR: How Do You Calculate the Real Cost Per Token for LLM Inference?
- Formula: cost per million tokens (CPM) = (GPU cluster $/hour) ÷ (measured tokens/sec × 3,600 ÷ 1,000,000). Plug in your own measured throughput, not a vendor's benchmark.
- Quantization: FP8 KV-cache quantization on Llama-3.1-8B lifts H100 output throughput 14.9% and cuts total runtime 13.0% versus BF16, as of vLLM's 22 Apr 2026 benchmark.
- Context length: Llama 3 70B's KV cache grows from 0.313GB at 1,000 tokens of context to 39.06GB at 128,000 tokens, a 125x jump that shrinks how many concurrent requests a fixed GPU budget can serve.
- Price: Spheron's H100 GPU rental runs $2.65/hr on-demand as of 18 Sep 2026, the $/hour half of the formula this post walks through.
The LLM API Cost Comparison Formula: From Hourly GPU Rate to Dollars Per Million Tokens
The formula has three inputs and one output, and it's the same math every cost comparison on this blog eventually runs:
CPM ($ per million tokens) = (cluster $/hour) ÷ (tokens/sec × 3,600 ÷ 1,000,000)
- Cluster $/hour is the full hourly cost of every GPU the model needs to run, not the per-GPU rate. A 70B model at FP16 needs roughly two H100s to fit in VRAM, so the numerator is two times the hourly rate, not one.
- Tokens/sec is the sustained output rate you measure from your own serving stack, under your own traffic pattern, at your own batch size. This is the term almost every public cost comparison gets wrong, and it's the subject of the next section.
- 3,600 ÷ 1,000,000 converts tokens/sec into tokens/hour, then onto a per-million-token basis, so the result is comparable across models, GPUs, and providers.
Get the tokens/sec term right and the rest is arithmetic. Get it wrong, and a spreadsheet full of correct-looking division still produces a number nobody else can reproduce. Our AI inference cost economics playbook runs this same formula inside a full GPU FinOps model; this post stays narrowly on the tokens/sec term, because that's where the formula actually breaks in practice.
Why the Vendor's Tokens/Sec Number and Our Measured Number Don't Match
A GPU vendor's tokens/sec headline and the number your API actually sustains under real traffic measure two different things, not the same thing at two different levels of optimism. The vendor number usually comes from an offline, latency-unconstrained batch benchmark built to find a ceiling. Your production number comes from a latency-constrained server balancing throughput against how long a user will wait for a first token.
Offline/Batch Benchmarks vs Latency-Constrained Serving
That's the opposite of how a production LLM API actually serves users, and it's exactly the scenario most published GPU throughput records come from.
Artificial Analysis takes the opposite approach on purpose. If you're benchmarking your own deployment, run it under the concurrency and latency budget you'll actually serve, the same way vLLM and TensorRT-LLM compare differently depending on whether the engine is tuned for raw throughput or for time-to-first-token.
How Batch Size Moves Cost Per Token by an Order of Magnitude
Batch size is the single biggest lever in the entire formula, because the denominator can move by close to two orders of magnitude while the numerator, the GPU's hourly rate, doesn't move at all.
Because the hourly rate cancels out of that ratio, throughput changes are cost changes too.
This is the mechanism continuous batching in vLLM exists to capture automatically: instead of a batch size a human sets once and forgets, iteration-level scheduling keeps refilling the batch with queued requests as others finish, pushing a live server's effective batch size toward the high-throughput end of the throughput curve without anyone deciding the number in advance. A server running near batch 1 most of the day because traffic is bursty will pay more per token; a server saturated with concurrent requests will pay less. The gap between the two is the entire economic argument for batching-aware serving.
How Quantization and Context Length Shift the Same Formula
Quantization and context length don't change the formula, they change one of its two inputs. Quantization raises tokens/sec at the same $/hour. Long context eats the concurrency that made a high tokens/sec number possible in the first place.
FP8 and INT4: More Tokens/Sec at the Same $/Hour
FP8 doesn't touch a GPU's hourly rate; it only touches the throughput term.
INT4 (AWQ- or GPTQ-style weight quantization) pushes the same lever further, at the cost of a quality evaluation you have to run yourself. There's no published, model-agnostic number for exactly how much INT4 buys without measuring your own accuracy regression first, so treat it as the next lever to test rather than a figure to assume.
Context Length: KV Cache Growth Cuts Concurrency, Raises Cost Per Token
Context length moves the other input. It doesn't lower tokens/sec directly, it shrinks how many concurrent requests fit in VRAM at once, which decides whether you're serving near the high-throughput end of the curve or back down near the low-throughput end.
The consequence for the formula: a model that looks cheap per token at an 8K-token benchmark can look considerably more expensive at 128K, not because the GPU got slower, but because the same VRAM budget now fits far fewer concurrent requests, pushing effective throughput back down. Any cost-per-token calculator that doesn't ask for your context length is missing the input that decides whether it's measuring a realistic batch size at all.
Turn This Into Your Own Cost-Per-Token Calculator
Building your own calculator takes the same three inputs as the formula above, measured against your own deployment instead of a vendor's benchmark:
- Pick the exact scenario you're pricing. Model, precision, input/output token split, and the context length you'll actually serve, since context length decides how much VRAM, and therefore concurrency, is available.
- Measure tokens/sec under your real concurrency and latency budget, not an offline batch run. Run your serving stack at the request rate and first-token latency target you'll actually commit to, and log the sustained output tokens/sec across that window, not the single-request or empty-queue number.
- Price the exact cluster your model needs, not a single GPU, if the model needs more than one to fit in VRAM. Multiply the per-GPU on-demand or spot rate by the GPU count.
- Divide. CPM = cluster $/hour ÷ (tokens/sec × 3,600 ÷ 1,000,000). Rerun it whenever the model, quantization, context length, or GPU rate changes, since any one of the four moves the answer.
Once you have your own measured number, Groq's API pricing versus GPU rental breakdown shows how to take that CPM figure and find the actual crossover point where renting the GPU yourself beats paying a managed API per token, rather than assuming self-hosting is cheaper by default.
For the $/hour half of the formula, Spheron's marketplace pricing aggregates on-demand and spot rates across 5+ providers with per-minute billing and no minimum commitment, so the number you plug in reflects what you'd actually pay for the exact volume you're testing, not a reserved-capacity rate you'd only get by committing to more than you need. It only supplies the numerator, though: the tokens/sec denominator still has to come from a benchmark you run yourself, since a pricing page has no way to publish your model's per-GPU throughput. For teams whose call volume sits below the self-hosting breakeven point either way, a managed API stays the cheaper option regardless of whose GPU rate goes into this formula.
The formula only works if the tokens/sec you plug in came from your own serving stack, not a vendor's peak-batch slide. Benchmark it on real hardware before you commit to a GPU count.
Frequently Asked Questions
Use the full cluster's hourly rate, not the per-GPU rate. A 70B model at FP16 needs roughly two H100s just to fit weights and KV cache, so the numerator in the cost-per-million-tokens formula is two times the per-GPU on-demand rate, not one. Divide that cluster rate by measured tokens per second, converted to a per-million-token basis, to get a defensible CPM figure. Spheron's on-demand H100 SXM5 rate is $2.65/hr as of 18 Sep 2026, so a two-GPU cluster's numerator is twice that figure.
Because the two figures usually come from different measurement scenarios, not different levels of optimism about the same one. A vendor's headline is almost always closer to aggregate system throughput at high batch; your production server, especially under light load, often sits closer to the per-user throughput experienced by individual requests. Check which scenario produced the vendor's number before assuming your own deployment is misconfigured.
There's a real trade-off, not a free lunch. Pushing more concurrent requests through the same GPU raises the aggregate throughput the cost-per-token formula divides by, but it also lengthens how long any single request waits for its next token, because more sequences are sharing the same forward pass. Increasing batch size raises the aggregate throughput (lowering cost per token) while lengthening per-user latency. Continuous batching narrows the trade-off by refilling the batch at the iteration level instead of waiting for a full batch to clear, but it doesn't erase it. A server tuned purely for the lowest cost per million tokens will feel slower to any individual user than one tuned for the lowest latency, so the right batch size depends on which side of that trade-off your SLA actually cares about.
No, the size of the gain depends on where in the stack you apply it and what's actually the bottleneck. Quantizing the KV cache produces a throughput lift at the identical hourly rate. Going further, to INT4 weight quantization with AWQ or GPTQ, pushes the same lever harder still, but there's no published, model-agnostic figure for how much it buys without measuring your own accuracy regression first, so treat it as the next lever to test rather than a number to assume.
