Continuous batching in vLLM, and the equivalent scheduling change in TGI and TensorRT-LLM, is the reason a rented GPU running any of the three does more work per hour today than the same GPU did in 2023. It's a scheduling change, not a hardware one, and it's why cost per token has fallen even as GPU rental prices have stayed roughly flat. This post stays at the mechanism level: what continuous batching is, why static batching wastes GPU time, and how vLLM's version compares to TGI's and TensorRT-LLM's, closing with what the utilization gain means in GPU-hours per million tokens. If you've already decided on vLLM and want the flag-by-flag tuning guide, PagedAttention's memory math, and chunked prefill for long-context traffic, that's a separate, deeper post: our continuous batching, PagedAttention, and chunked prefill deep-dive. This one is for picking a mental model and a serving engine; that one is for configuring the engine you picked.
What Continuous Batching in vLLM Means, in One Paragraph
Continuous batching is a GPU scheduling strategy for LLM inference that decides the running batch at every decode step instead of once per batch. When a request finishes generating, the scheduler frees its slot and immediately fills it with the next queued request, rather than holding that slot idle until every other request in the original batch also finishes. The practical effect is that the GPU rarely sits idle waiting on the slowest request in a batch, which is exactly the failure mode static batching has.
Static Batching's Dead-Time Problem
Static batching groups a fixed set of requests, sends them to the GPU together, and releases the whole batch only once every request in it has finished generating. That's simple to implement, and it's also where most of the wasted GPU-hours in early LLM serving stacks came from.
Why the Whole Batch Waits for the Slowest Request
LLM output lengths vary a lot within a single batch. One request might stop after 40 tokens because the model hit an end-of-sequence token early; another in the same batch might run for 600 tokens. Under static batching, the 40-token request's GPU slot doesn't get reassigned. It sits idle, computing nothing, until the 600-token request finally finishes and the whole batch is released together.
Multiply that across a batch of 16 or 32 concurrent requests with realistic output-length variance, and a large share of every batch's total GPU-seconds is spent on requests that have nothing left to do.
How Continuous Batching Fills the Gaps
Continuous batching removes the batch-level lock entirely. Instead of a fixed group of requests running together from start to finish, each GPU "slot" in the scheduler is treated independently. The moment a slot's request completes, it's replaced.
Static batching, one GPU slot per request, batch releases together:
Slot 1 (req A, runs full-length): ██████████████████
Slot 2 (req B, finishes early): ██████░░░░░░░░░░░░ <- idle from here to batch end
Slot 3 (req C, finishes early): ████░░░░░░░░░░░░░░ <- idle from here to batch end
Slot 4 (req D, finishes early): ██░░░░░░░░░░░░░░░░ <- idle from here to batch end
Continuous batching, same 4 slots, refilled the moment they free up:
Slot 1: ██████████████████ (A, runs full-length)
Slot 2: ██████[E]██████[?] (B done -> E admitted immediately, no gap)
Slot 3: ████[F]████[G]███ (C done -> F admitted; F done -> G admitted)
Slot 4: ██[H]██████[I]██ (D done -> H admitted; H done -> I admitted)In the static case, three of four slots go idle (░) for most of the timeline, and that idle capacity is simply lost, the batch won't release until Slot 1 finishes. In the continuous case, every ░ gets replaced by a new request the instant a slot opens, so the GPU has almost no idle time once the queue is deep enough to keep every slot fed.
Iteration-Level Scheduling, Step by Step
At each decode step, the scheduler does three things in order:
- Check which sequences in the running batch finished on the previous step (hit an end-of-sequence token or a max-length cap).
- Free those sequences' resources (the compute slot and, in vLLM's case, the PagedAttention KV cache blocks tied to that sequence).
- Admit the next queued request into the freed slot, subject to memory availability, and include it in the current step's forward pass alongside the requests still generating.
This repeats every single decode step, not once per batch. That's the "iteration-level" part: the batch composition is a moving target the scheduler re-decides constantly, rather than a fixed group locked in at request time.
Continuous Batching in vLLM, TGI, and TensorRT-LLM
All three major open-source serving engines implement continuous batching. The mechanism is close to identical across them; what differs is defaults, tuning surface, and what it's paired with.
| Engine | Name for the technique | Default state | Paired with |
|---|---|---|---|
| vLLM | Continuous batching | Always on, no disable flag | PagedAttention KV cache |
| Hugging Face TGI | Continuous batching | Always on | Manual prefill/total-token tuning |
| NVIDIA TensorRT-LLM | In-flight batching | Always on in the executor API | Compiled, kernel-optimized engines |
vLLM: On by Default, Paired with PagedAttention
vLLM has no setting to turn continuous batching off. A vLLM maintainer confirmed this directly in a GitHub discussion: "this is enabled by default and cannot be turned off. Turning off continuous batching requires a rewrite of our system architecture, which also brings no benefit in performance." The two levers you do control are --max-num-seqs, which caps how many sequences the scheduler can run concurrently, and --max-num-batched-tokens, which caps the total tokens processed in a single step.
Continuous batching only solves the compute-scheduling half of the problem. The other half is memory: reserving KV cache space for every admitted sequence without fragmenting VRAM or over-allocating for sequences that never reach their max length. vLLM pairs continuous batching with PagedAttention, which allocates KV cache in fixed-size blocks on demand instead of reserving a sequence's full max-context-length upfront. Our own deep-dive on continuous batching, PagedAttention, and chunked prefill covers the memory math and full vLLM flag reference if you're tuning a production deployment rather than evaluating the concept, and Spheron's vLLM inference server setup guide covers installation, systemd service configuration, and the tuning flags once you're connected to an instance.
Hugging Face TGI: Continuous Batching
--max-batch-total-tokens is the overall token ceiling across a batch's prefill and decode combined.
If you're currently on TGI, our TGI migration guide walks through translating TGI flags and batching config to either replacement.
NVIDIA TensorRT-LLM: In-Flight Batching
TensorRT-LLM calls the same technique in-flight batching. NVIDIA describes the mechanism directly: "rather than waiting for the whole batch to finish before moving on to the next set of requests, the TensorRT-LLM runtime immediately evicts finished sequences from the batch" and begins executing new requests while others are still in flight. NVIDIA's own measurement is that "in-flight batching and the additional kernel-level optimizations enable improved GPU usage and minimally double the throughput on a benchmark of real-world LLM requests on NVIDIA H100 Tensor Core GPUs."
TensorRT-LLM's tradeoff versus vLLM and TGI is compilation. Engines have to be built ahead of time for a specific GPU, precision, and shape configuration, which buys faster steady-state throughput at the cost of a build step vLLM doesn't require. If you're deciding between the two, our vLLM vs TensorRT-LLM comparison has head-to-head throughput and TTFT numbers on the same H100, and the TensorRT-LLM production deployment guide covers the engine-build and multi-GPU serving steps in depth.
Measured Throughput Gain on Our Own Fleet
Here's what the difference looks like in practice, not just in a timeline diagram. We benchmarked static batching against vLLM's continuous batching (v0.18.0) serving Llama 3.3 70B FP8 on an H100 SXM5 80GB, scaling concurrency from 4 to 128 requests. The full run, including the 16-, 32-, and 64-concurrency rows, is in our companion deep-dive's benchmark table.
The shape is what you'd expect from the mechanism: static batching's utilization holds flat regardless of queue depth, since adding more requests just means more batches processed one after another, each with the same idle tail. Continuous batching's utilization scales with concurrency instead, climbing as the queue gets deep enough to keep every slot fed.
What It Means for GPU-Hours per Million Tokens
Utilization percentages are useful for diagnosing a deployment. They're less useful for a budget. The number that actually matters when you're deciding how many GPUs to rent, and for how long, is GPU-hours consumed per million output tokens served, and on a decode-bound workload that number scales with utilization: get twice as much useful work out of every GPU-second, and you need roughly half the GPU-hours for the same token volume.
On the H100 SXM5 benchmark referenced above, static batching's utilization stayed flat in the 30-40% band no matter the concurrency, while continuous batching's climbed with it, from about 62% at 4 concurrent requests to about 87% at 128. Taking those endpoints at face value, switching to continuous batching cuts GPU-hours needed for the same token volume by roughly 1.6x to 2x at the lower-concurrency end (62 ÷ 40 and 62 ÷ 30) and by roughly 2.2x to 2.9x at the higher-concurrency end (87 ÷ 40 and 87 ÷ 30), before you've touched hardware, quantization, or anything else. Concretely: a deployment that needs 100 GPU-hours a day to hit its token volume under static batching needs roughly 35 to 63 GPU-hours under continuous batching for the same output, depending on where in that concurrency range it actually runs.
Multiply either end of that range by an hourly rate and it turns into a dollar figure a buyer can act on. On Spheron's marketplace, the cheapest current on-demand H100 80GB listing is a single-GPU instance at $2.64/hr. At 100 GPU-hours a day, that's $264 under static batching against roughly $92 to $166 under continuous batching for the same token volume, purely from the scheduler.
Pricing fluctuates based on GPU availability. The price above is based on 01 Sep 2026 and may have changed. Check current GPU pricing → for live rates.
None of this requires a different GPU cloud vendor to capture. It's a serving-engine default, not a hardware feature. Where the choice of GPU cloud actually matters is when you want to test scheduler settings, like --max-num-seqs or --enable-chunked-prefill, without committing to a long rental first. Spheron aggregates 5+ providers with per-minute billing and no minimum term, and its on-demand and spot instances give bare-metal access to the box, so vLLM, TGI, and TensorRT-LLM flags are yours to set directly rather than hidden behind a managed inference API's fixed configuration. That control matters for tuning; it matters less if you're already committed to a fully managed endpoint like Bedrock or Together AI, where continuous batching runs by default and you never touch a flag at all, at the cost of not being able to change one either.
If you're still deciding which serving engine fits your workload rather than how to configure the one you've picked, our vLLM vs TensorRT-LLM vs SGLang decision framework breaks the choice down by bottleneck instead of by benchmark leaderboard position. And if "batching" in your workload actually means offline, queue-based processing rather than live request scheduling, our batch inference guide covers that different problem: the scheduler behavior in this post is about concurrent online requests, not offline job queues.
Continuous batching is table stakes now, on by default in every engine worth deploying. The GPU-hours-per-million-tokens math above is what to check next: whether your current concurrency, KV cache headroom, and request-length distribution are actually letting your engine's scheduler do its job, or whether something else, like a KV cache pool sized too small to hold enough concurrent requests, is quietly capping you below the utilization this post describes.
Continuous batching is already doing the scheduling work; what it needs is a GPU you can tune without waiting on a long commitment.
Frequently Asked Questions
Because vLLM's engine is built around iteration-level scheduling from the ground up rather than bolted on as an option. The scheduler re-evaluates the running batch at every decode step, evicting completed sequences and inserting queued ones in their place, and a vLLM maintainer has said turning it off would require rewriting the system architecture for no performance benefit. There's simply no flag for it.
They're often used loosely to mean the same idea, but continuous batching specifically means re-forming the batch at every decode step (iteration-level scheduling). Static batching locks a group of requests together until the longest one finishes, so a finished request's GPU slot sits idle instead of picking up the next request in queue.
Yes, under the name in-flight batching. NVIDIA's runtime evicts finished sequences from the batch and starts new requests without waiting for the full batch to complete, the same mechanism vLLM and Hugging Face TGI use under different names.
