Chunked Prefill vLLM Explained: How It Smooths TTFT Under Load (2026)

Ask vLLM or SGLang to serve a 30,000-token RAG prompt alongside a dozen short chat turns, and the scheduler has to decide which one goes first. Chunked prefill is the setting that makes that call for you, and in vLLM's current engine it isn't really a setting at all. It's been the scheduler's default behavior since the V1 engine replaced V0, and SGLang ships it at a fixed chunk size out of the box. Almost nobody running either engine has measured what it actually does to time-to-first-token (TTFT) once real concurrent traffic shows up, so here's the mechanism, the TTFT-versus-ITL knob each engine still exposes, and what it buys on hardware running production traffic today.

TL;DR: How Chunked Prefill Smooths TTFT Under Load

• Mechanism: chunked prefill splits a long prompt's prefill into token-capped chunks, letting decode steps from other requests run between them instead of one blocking forward pass.
• vLLM default: vLLM's V1 scheduler enables chunked prefill by default whenever possible and no longer separates prefill from decode, so there's no clean disable flag.
• SGLang default: --chunked-prefill-size defaults to 8192 tokens; setting it to -1 is SGLang's off switch.
• Tradeoff: vLLM's chunked-prefill RFC measured roughly 10-20% better latency at sustainable QPS and near 2x at high QPS.
• Spheron: B200 instances serving open-weight models under concurrent load kept TTFT between 0.4s and 1.9s at 32 sessions, per Spheron's Oct 2026 fleet data. Compare live GPU pricing.

The Problem: Long Prefills Block Decode Steps

A request to an LLM runs through two phases with opposite shapes. Prefill processes the entire input prompt in one pass, attending every token against every other token, and that's a compute-bound dense matrix multiply. Decode generates output tokens one at a time, reading the accumulated key-value cache from memory for each new token, and that's bandwidth-bound. Our continuous batching explainer covers how vLLM's iteration-level scheduler fills idle decode slots the moment a request finishes, which solved the throughput half of batching.

It didn't solve this half: continuous batching re-forms the running batch at every decode step, but if one of the requests entering that step is a long prompt's prefill, the scheduler still has to run that entire prefill as one forward pass before it can move on. Every other request sharing the GPU that step waits behind it. A short chat message queued right after a 20,000-token document upload gets stuck behind that document's prefill, even though continuous batching is working exactly as designed. This is head-of-line blocking, and it's a latency problem, not a throughput one, which is why fixing it needed a different mechanism than the one that fixed utilization.

How Chunked Prefill Splits the Work

Chunked prefill splits a long prompt's prefill into fixed-size pieces, capped by a token budget per scheduler step, so decode steps for other active requests can run between chunks instead of waiting out one uninterrupted forward pass. A 30,000-token prompt no longer claims the GPU in a single multi-hundred-millisecond burst; it gets processed across several scheduler iterations, with other requests' decode steps slotted in between.

The lever that sets the chunk size is --max-num-batched-tokens in vLLM. Lower values, around 2,048 tokens, improve inter-token latency (ITL) for everyone else in the batch, because fewer prefill tokens compete with decode steps on each scheduler iteration. Higher values, 8,192 and up, improve the long prompt's own TTFT, because more of its prefill gets processed per step and it reaches its first output token sooner. That's the actual tradeoff this setting controls: your choice doesn't eliminate latency cost, it decides who pays it, the long request's own TTFT or everyone else's decode smoothness.

This isn't a purely theoretical tradeoff either. Sarathi-Serve, the research paper that paired chunked prefills with what it calls stall-free batching, reports up to 2.6x higher serving capacity on Mistral-7B running on a single A100 compared to vLLM's unchunked scheduler, and up to 5.6x higher end-to-end serving capacity on Falcon-180B with pipeline parallelism. That's capacity, not just smoother percentiles: the same GPU fleet handling meaningfully more concurrent load once prefill stops monopolizing scheduler steps.

vLLM vs SGLang: How Each Implements Chunked Prefill

Both major open-source serving engines ship chunked prefill, but only one still lets you turn it off.

Chunked prefill started life as an optional scheduling policy. It was proposed in GitHub RFC #3130, "Upstream Chunked Prefill," before being merged and later made mandatory in the V1 scheduler. The RFC's own author, rkooo567, noted that TTFT is supposed to be a bit slower because chunked prefill adds per step overhead and KV cache memory load overhead. The original unchunked default was tuned to get one request's first token out as fast as possible; chunked prefill trades a little of that away in exchange for every other concurrent request staying responsive. The same author's broader read on when that trade pays off: "chunked prefill greatly improves latency when QPS is high, but has competitive performance at low QPS." The numbers behind that claim, from the same RFC thread: roughly 10-20% better latency at sustainable QPS, close to 2x better at high QPS.

V1 then removed the choice. A vLLM forum explainer confirms chunked prefill cannot be turned off in V1, because the scheduler no longer distinguishes prefill and decode as separate phases at all, there's just one unified step loop. vLLM's own optimization docs add the one place the old on/off framing still surfaces: if chunked prefill is disabled, max_num_batched_tokens must exceed max_model_len or vLLM can fail at startup, a legacy constraint from before V1 made the setting moot.

SGLang's server arguments reference keeps the lever vLLM dropped. --chunked-prefill-size caps tokens processed per chunk and defaults to 8,192; setting it to -1 disables chunking entirely, processing each prompt's prefill in one pass. If your workload genuinely never mixes long and short prompts, SGLang lets you test the unchunked path directly. vLLM doesn't give you that comparison anymore. For the deeper engine-vs-engine throughput picture beyond this one knob, our vLLM vs SGLang benchmark guide covers RadixAttention's prefix-cache advantage on top of this.

Measured Effect on TTFT Under Concurrent Load

None of the above tells you what chunked prefill actually buys once a node is carrying real concurrent traffic, so here's what Spheron's own fleet showed serving several current open-weight models in October 2026, with each engine's default chunked-prefill scheduler running in the background, no special tuning applied.

What's notable isn't just that TTFT stayed under 2 seconds at 32 concurrent sessions across every model and GPU split. It's the volume moving through the scheduler while it held there: the GLM-5.3-Flash node's 4-GPU copies each read 151,270 prompt tokens per second and wrote 1,873 output tokens per second under those 32 agent sessions with zero errors, and DeepSeek V4 Flash's smaller 2-GPU copies each handled 137,509 prompt and 1,607 output tokens per second, also with zero errors. That's the chunked-prefill scheduler doing its job: tens of thousands of prefill tokens per second arriving continuously without pushing any session's TTFT past a couple of seconds.

The B200-versus-B300 pattern across both Kimi K2.7 and GLM-5.3 is also worth sitting with: splitting an 8xB300 node into two 4-GPU copies gave lower per-user token speed than running one 8-GPU copy on 8xB200 (254 vs 324 tok/s for a single GLM-5.3 user), but in exchange the B300 node served twice the concurrent agent sessions, 64 against 32. Smaller per-copy footprints shrink the queue each chunked-prefill scheduler has to manage, which is a big part of why those B300 TTFT figures came in lower than B200's in the same table. Read against the TTFT and ITL latency budget framework, these are the kind of numbers you'd plug in to decide whether a 400ms or a 2-second TTFT budget is realistic for a given node split before you commit capacity to it.

Chunked Prefill vs Disaggregated Prefill

Chunked prefill and disaggregated prefill solve the same symptom, head-of-line blocking from long prompts, with opposite amounts of hardware. Chunked prefill is a scheduling trick on a single node: it slices one GPU's prefill work into smaller pieces so decode for other requests can slot in between, no extra hardware, no network hop. Disaggregated prefill is a topology decision: prefill and decode run on separate, differently-provisioned GPUs or nodes entirely, with the KV cache shipped from the prefill node to the decode node over the network once the prompt finishes processing.

That separation buys real throughput when it fits: our prefill-decode disaggregation guide measured roughly double the throughput of two colocated H100s by splitting prefill onto a compute-dense GPU and decode onto a memory-dense one. But it costs a network transfer and a second class of node to provision, monitor, and keep balanced. Chunked prefill costs neither. That's why it's the right first move for most deployments: it needs no topology change, works inside the single-node setup you already have, and only stops being enough once your prompts are consistently long and your concurrency is high enough that one node genuinely can't do both jobs at once anymore.

Which GPU and Config to Pick for Latency-Sensitive Serving

Chunked prefill is already running by default on whatever you deploy, so the actual decision in front of you is how to split the node and where to set --max-num-batched-tokens (or --chunked-prefill-size on SGLang) for your specific latency budget.

If your priority is the fastest possible response for each individual user and you can tolerate a lower session ceiling, one 8-GPU copy on a B200 node, matching the Kimi K2.7 and GLM-5.3 configurations above, gets you the higher per-user token rate. If your priority is serving more concurrent agent sessions at a given latency budget, splitting an 8xB300 node into smaller copies traded per-user speed for roughly double the session count in the GLM-5.3 comparison above, and came with lower TTFT in the bargain. Neither choice is universally correct; it's the same TTFT-versus-ITL tradeoff chunked prefill's chunk-size knob makes internally, just applied one layer up at the fleet-provisioning level instead of the scheduler-config level.

For the full flag-by-flag reference, including the PagedAttention memory math that interacts with how much headroom you have to raise --max-num-batched-tokens, our continuous batching, PagedAttention, and chunked prefill deep-dive covers that configuration ground in depth; this post stays at the mechanism and the buying decision.

Testing that tradeoff needs a GPU you actually control. A managed endpoint like Bedrock or Together AI gives you continuous batching and usually chunked prefill for free, fine if you never need to touch either. It also means you can't set --max-num-batched-tokens or --chunked-prefill-size at all, since those knobs live inside a serving stack you don't configure. Spheron's pricing page offers on-demand, spot, and reserved GPU instances billed per minute with full root access to the box, across H100, H200, B200, B300, and more, which is what lets you run that A/B yourself, chunk size by chunk size, before committing to a fleet shape.

Chunked prefill is already smoothing TTFT on whatever node you're running; the question left is whether that node's split actually matches your latency budget.

Check B200 GPU pricing →