Double the batch size, double the throughput. It's the same instinct that makes people assume doubling GPU count doubles training speed, and it's wrong for a different reason here. GPU inference throughput scaling runs into two separate ceilings on the exact same chip: how fast it can compute, and how much memory is left to hold every in-flight request's KV cache. This post walks through a real batch size sweep and which ceiling you're actually more likely to hit.
TL;DR: GPU Inference Throughput Scaling vs Batch Size
- Not linear. Databricks measured MPT-7B throughput on a single A100 rising 14x (0.9 to 12.5 req/sec) over a 64x batch jump, but decelerating hard: +5.9x for the first 8x, only +1.19x for the last 2x.
- KV cache usually caps it first. 128 concurrent requests at a 4K-token context need roughly 128x one request's KV cache memory, and that runs out before compute does.
- Spheron bills per minute with no minimum term, so a batch-size sweep on a rented H100 costs a few dollars as of 15 Sep 2026. Compare GPU pricing.
The Common Assumption: More Requests in Flight, Proportionally More Tokens Out
The mental model is easy to reach for because it matches how a lot of infrastructure actually behaves. Add a second web server, roughly double your request handling capacity. Add a second database replica, roughly double your read throughput. So the assumption carries over cleanly: if one request in flight produces X tokens per second, sixty-four requests in flight should produce something close to 64X.
It's a reasonable starting point, and it's wrong for LLM inference in a way that matters for how you size a deployment. The mechanism isn't communication overhead between machines, the way it is when teams assume doubling GPU count doubles training throughput. This is a single GPU, no network hop, no synchronization barrier between chips. The ceiling here comes from something narrower: how the GPU's own compute units and memory bandwidth trade off against each other as more sequences share the same forward pass.
Autoregressive decoding generates one token per sequence per step, and every step has to read the full set of model weights from VRAM before it can compute anything. At batch size 1, that read happens once and produces one token's worth of work, which is a bad trade. Our memory wall post covers why this makes single-request decode memory-bound on almost every GPU: for a 70B model at FP16, that's roughly 140 GB of weights moved per token step, and the transfer time sets a hard floor on latency that adding compute FLOPS can't touch. Batch size is the one lever that changes this equation, because it lets multiple requests share that same weight read. The question this post answers is how far that sharing actually goes before something else runs out.
What the Numbers Show: GPU Inference Throughput Scaling from Batch Size 1 to 128
Databricks ran exactly this sweep on a single A100 serving MPT-7B, and the shape of their published curve is the clearest public illustration of how GPU inference throughput scaling actually plays out on real hardware. Throughput climbed from 0.9 requests per second at batch size 1 to 5.3 at batch size 8, 8.0 at batch size 16, 10.5 at batch size 32, and 12.5 at batch size 64, a 14x throughput gain across a 64x increase in batch size, while per-request latency rose only 4x over the same range.
| Batch size | Throughput (req/sec) | Throughput gain vs batch 1 |
|---|---|---|
| 1 | 0.9 | 1x (baseline) |
| 8 | 5.3 | ~5.9x |
| 16 | 8.0 | ~8.9x |
| 32 | 10.5 | ~11.7x |
| 64 | 12.5 | ~13.9x |
Source: Databricks LLM inference performance engineering benchmarks, MPT-7B on a single A100.
Look at the shape, not just the endpoints. Between batch 1 and batch 8, throughput jumps 5.9x for an 8x batch increase, nearly linear. Between batch 32 and batch 64, throughput only gains another 1.19x for a 2x batch increase. That deceleration is the tell that this workload is approaching a ceiling well before batch 64, and it's the same deceleration you'll see if you run this sweep yourself on any model and GPU pair.
Databricks' own published sweep stops at 64, but the deceleration doesn't reverse itself if you keep pushing to 128 and beyond. It just continues until one of two hard ceilings stops it outright. We've mapped the arithmetic-intensity math connecting batch size to a GPU's compute-versus-memory ridge point in detail elsewhere, with real H100 FLOPS and bandwidth numbers, so this post won't re-derive it. The short version that matters here: crossing that ridge point takes a much higher batch size than most interactive serving workloads ever reach, which is exactly why a second, lower ceiling, KV cache memory, is the one that actually bites in most production deployments.
Where It Plateaus (and Why KV Cache, Not Compute, Is Usually the Ceiling)
A serving stack hits one of two ceilings as batch size grows, and knowing which one you're approaching determines whether the fix is a different GPU, a different quantization scheme, or just a config change. vLLM's own performance model names the batch size where the transition happens the saturation batch size, or B_sat: below it, step time is dominated by HBM bandwidth and stays nearly flat regardless of batch size, and beyond it, kernels become compute-bound and step time grows roughly linearly with batch size. As the vLLM engineering team put it, that's the point where adding more concurrent requests stops being free.
The Compute Ceiling: Real, but Further Out Than Most Workloads Reach
The compute ceiling is the one the batch-size-scales-throughput assumption implicitly expects, and it's real, it's just further out than most workloads ever reach. Every GPU has a point on its performance curve where memory bandwidth and peak FLOPS become equally limiting. Below it, more compute headroom doesn't help because the bottleneck is data movement. Above it, more memory bandwidth doesn't help because the bottleneck is the math itself.
Databricks' own engineering team put the consequence plainly in their benchmarks post: "After a certain batch size, i.e., when we cross to the compute bound regime, every doubling of batch size just increases the latency without increasing throughput." That's the ceiling you hit if nothing stops you first, and it comes later for a smaller model or a lower-precision format than for a larger model running at FP8.
TensorRT-LLM's own tuning guidance treats this as a curve to find, not a limit to maximize past. The recommendation is to locate the knee of the throughput curve where additional batching provides diminishing returns, and set max_batch_size roughly 20-30% above that point rather than pushing it as high as possible. That's a direct acknowledgment that the curve bends well before it's technically capped, and that overshooting the knee mostly just adds latency for no real throughput gain, which lines up with the Databricks A100 sweep's own deceleration between batch 32 and 64.
The Memory Ceiling: When KV Cache Runs Out Before Compute Saturates
For most production serving stacks, this is the ceiling that actually bites, and it bites earlier than the compute ceiling. Every additional request in the batch needs its own key-value cache: a growing block of memory that holds the attention keys and values for every token in that request's context so far. Unlike model weights, which are fixed size and shared across the batch, KV cache scales linearly with both batch size and context length. A batch of 128 requests each carrying a 4K-token context reserves roughly 128 times the KV cache memory of a single request at the same context length, on top of whatever's already allocated to the model's weights.
TensorRT-LLM ships with a default max_batch_size of 256, and production deployments chasing maximum throughput often raise it to 2048, but that setting only pays off if KV cache memory actually has room to support that many concurrent sequences at your context length. Raise the batch size ceiling without the VRAM to back it and you get evictions, request queuing, or out-of-memory failures rather than the throughput gain the higher number implies. This is why our KV cache compression coverage of Google's TurboQuant work matters more than another GPU generation for a lot of teams: compressing KV cache doesn't move the compute ceiling at all, but it directly raises the batch size at which memory runs out, which is the ceiling most deployments actually hit.
Here's the diagnostic difference in practice. If you profile a serving workload and see SM (compute) utilization climbing toward 90%+ while GPU memory utilization has headroom left, you're approaching the compute ceiling and TensorRT-LLM's knee-of-the-curve guidance applies directly. If you see GPU memory nearly full with SM utilization well under that, you're hitting the KV cache ceiling, and the fix is a smaller context window, a quantized KV cache, or a GPU with more VRAM, not a faster GPU. Our continuous batching explainer covers the scheduling side of this same problem: continuous batching fills idle GPU slots as requests finish, but it can only fill a slot that has KV cache memory available to allocate to it, so a KV-cache-constrained deployment hits its ceiling regardless of how good the scheduler is.
The Buying Consequence: Sizing Concurrency Instead of Just Renting a Bigger GPU
The practical upshot is that "rent a faster GPU" is the wrong first move for most throughput problems, because most throughput problems are memory-capacity problems, not compute problems. A GPU with more FLOPS doesn't help if you're running out of VRAM for KV cache at batch size 40; a GPU with more VRAM and more memory bandwidth usually does.
The sizing exercise that actually matters is running your own sweep, on your own model, at your own target context length, because the saturation batch size and the KV cache ceiling are both specific to that combination, not to the GPU alone. That's a cheap enough experiment to run directly rather than estimate: launch batch sizes of 1, 8, 16, 32, 64, and 128 against your actual model and log throughput and GPU memory utilization at each step, the same shape as the sweep above. Spheron's per-minute billing with no minimum rental period means that sweep costs whatever GPU-hours it actually takes to run, not a monthly commitment to find out where your curve bends; H100 SXM5 on Spheron is currently listed at $2.98/hr on-demand and $2.05/hr on spot. Bare-metal access on Spheron also means vLLM's --max-num-seqs and --max-num-batched-tokens, or TensorRT-LLM's max_batch_size, are settings you can tune directly on the instance rather than parameters hidden behind a managed API's fixed configuration.
That said, Spheron doesn't run this benchmark for you, and renting a GPU on any provider doesn't change where a given model's curve bends. The KV cache ceiling is a function of model size, context length, and quantization scheme, not of which cloud a GPU comes from. What changes is how cheaply you can find that bend and how directly you can act on it once you have. If you're still deciding which serving engine to run the sweep on in the first place, our vLLM vs TensorRT-LLM vs SGLang benchmarks compare all three on the same H100, and our inference optimization decision framework walks through picking a stack before you start tuning batch size at all. And if what you're actually running is an offline job rather than a live serving endpoint, note that "batch" means something different there. Our batch LLM inference guide covers offline batch jobs, which is a distinct cost problem from the batch-size-per-request scaling covered here.
For a cost lens on the same curve, throughput per batch size converts directly into cost per token, and the same deceleration that shows up in requests-per-second shows up as diminishing cost savings once you're past your saturation batch size. Our GPU cost per token benchmarks run that conversion across seven models and multiple GPU types, which is the next step once you know where your own throughput curve bends.
Pricing fluctuates based on GPU availability. Spheron rates above are live as of 15 Sep 2026; other providers reflect their most recent published rates and may have changed. Check current GPU pricing → for live rates.
Finding your own saturation batch size takes an hour and a few dollars of GPU time, not a procurement cycle.
Frequently Asked Questions
Yes, but the gain shrinks with every doubling, it doesn't hold steady. In Databricks' MPT-7B sweep on a single A100, throughput reached 5.3 requests/sec at batch size 8, 8.0 at batch size 16, and 10.5 at batch size 32, each doubling adding less than the one before it. That decelerating curve, not a fixed ratio, is what signals you're approaching a saturation point where the bottleneck shifts to either GPU compute or KV cache capacity.
Two separate ceilings on the same GPU. Below the saturation batch size, decode steps are memory-bandwidth bound and step time barely moves as batch size grows, so throughput scales well. Once arithmetic intensity crosses the GPU's roofline ridge point, kernels become compute-bound and step time starts growing roughly linearly with batch size, which is when doubling batch size stops doubling throughput. In most real deployments, KV cache memory runs out and forces this ceiling before the compute ceiling is ever reached.
The mechanism is arithmetic intensity: FLOPs performed per byte of memory moved, which climbs as batch size grows because the same model weights get reused across more sequences in a single read. It only crosses a GPU's compute-versus-memory crossover point at batch sizes well beyond what most interactive serving ever reaches, which is why KV-cache-constrained is the far more common diagnosis than compute-bound. Watching GPU memory utilization against SM (compute) utilization while the workload runs at your real batch size tells you which one you're actually hitting.
It's the point where the memory reserved for active requests' key-value tensors runs out before the GPU's compute units saturate. TensorRT-LLM's default max_batch_size is 256, and production deployments chasing maximum throughput often raise it to 2048, but that setting only pays off if KV cache memory has room to actually support that many concurrent sequences at your context length. Past that point, adding batch capacity just triggers evictions or out-of-memory errors instead of more throughput.
Only if you've confirmed you're compute-bound, which is rare in production serving. Most deployments hit the KV cache ceiling first, and a bigger GPU with the same VRAM-per-dollar ratio doesn't move that ceiling much. Renting a GPU with more memory bandwidth and VRAM (H200 over H100, for example) or applying KV cache compression usually moves the needle further than raw compute upgrades.
