Engineering

vLLM 0.33 Release Notes: Speculative Decoding Drops Reserved Slots

Back to BlogWritten by Published Nov 16, 2026
vLLM 0.33 Release NotesvLLM 0.33 Speculative DecodingReserved Draft SlotsDSpark vLLMDFlashvLLM SchedulerSpeculative DecodingGPU Cloud
vLLM 0.33 Release Notes: Speculative Decoding Drops Reserved Slots

The vLLM 0.33 release notes list new model support and a CUDA bump, but the line worth reading twice is PR #59468: "Don't reserve DFlash/DSpark draft slots in the scheduler token budget." On the PR's own benchmark, fixing it nearly tripled DeepSeek-V4.1-Flash throughput on one node.

TL;DR: What Changes in vLLM 0.33 for Speculative Decoding

PR #59468 stops vLLM's scheduler from charging DFlash and DSpark drafts against the running-request token budget for slots they never consume, letting more requests run concurrently on the same node.

  • The number: output throughput on the PR's own benchmark went from 1,373 to 3,390 tok/s on a DeepSeek-V4.1-Flash node, from freed concurrency, not a faster drafter.
  • No accuracy change: greedy outputs on 20 fixed prompts were byte-identical before and after the fix.
  • Scope: applies to DFlash on both model runners and DSpark on MRv2 only; EAGLE-3, P-EAGLE, PARD, and plain draft models were never affected.
  • Spheron: the PR's own benchmark ran TP4 across 4 GPUs on one GB300 node; Spheron sells B300 on-demand and spot in multi-GPU configurations today, so that 4-GPU shape is reproducible without a reservation.

Speculative Decoding Explained: Why Draft Slots Exist in the First Place

Speculative decoding splits generation into two models working together: a small, cheap drafter proposes several candidate tokens ahead, and the expensive target model verifies all of them in a single forward pass instead of generating them one at a time. When the target model accepts a run of drafted tokens, the engine just advanced several positions for the cost of one verification step; when it rejects one, decoding falls back to the target model's own token and the drafter tries again from there. That verification step, speculative sampling, is what keeps the output distribution identical to plain decoding: the target model isn't trusting the draft, it's checking it.

Multi-token prediction (MTP) is a different mechanism that gets confused with speculative decoding because it produces the same kind of speedup. MTP bakes extra prediction heads into the base model itself, trained jointly with it, so one forward pass yields several tokens with no separate drafter and no verification step at all. DeepSeek ships models with an MTP-1 head by default for exactly this reason, and DSpark, DeepSeek's confidence-scheduled speculative decoding method, is designed to beat that MTP-1 baseline by stacking a real drafter on top of it, not to replace the concept.

EAGLE speculative decoding is the autoregressive end of the drafting spectrum: EAGLE-3's lightweight head proposes one token at a time, trained on the target model's fused internal features, with a fixed draft depth set once at server launch. DFlash and DSpark sit at the other end, drafting a whole block of tokens in one shot instead of a token chain. That block-drafting shape is exactly what made them eligible for the scheduler bug PR #59468 fixes, since EAGLE-3's draft head never asked the scheduler to reserve a separate block of slots the way a block drafter does.

Either style of drafter has to live inside vLLM's continuous-batching scheduler, which decides each step how many requests can run at once against a fixed token budget. That budget is the resource PR #59468 is actually about.

The Bug: DFlash and DSpark Were Reserving Slots They Never Used

Before the fix, vLLM's scheduler charged every DFlash request K slots and every DSpark request K-1 slots against its running-request token budget, where K is the drafter's block size. That charge assumed the draft tokens would land as extra rows in the target model's batch, the way an autoregressive drafter's proposals do. DFlash and DSpark don't work that way: both run their draft pass in its own separate buffer and hand the target model a block to verify, so neither one actually adds rows to the target batch at all.

The scheduler was reserving capacity for work that was never going to show up. On a server running near its token-budget ceiling, that meant fewer concurrent requests got admitted than the GPU could actually serve, not because memory or compute ran out, but because the accounting assumed a cost that didn't exist. The PR closes issue #52922, filed against exactly this mismatch between the reserved charge and the drafters' real behavior.

The Fix in PR #59468: Reserved Draft Slots vs the New Shared Allocation

The fix sets the draft-slot charge to 0 for both DFlash and DSpark and replaces it with a cap on running requests of max_num_batched_tokens // num_lookahead_tokens, keeping each drafter's separate pass inside its own buffers instead of borrowing room from the scheduler's shared budget.

DrafterOld scheduler chargeNew scheduler charge
DFlash (both model runners)K slots reserved per request0, capped by max_num_batched_tokens // num_lookahead_tokens
DSpark (MRv2 only)K-1 slots reserved per request0, same cap
EAGLE-3, P-EAGLE, PARD, plain draft modelsUnchangedUnchanged

Scope matters here: the fix lands for DFlash on both of vLLM's model runners, but for DSpark only on the newer MRv2 runner. If you're running DSpark on the older runner path, this particular fix doesn't reach you yet.

The Numbers: 1,373 to 3,390 Tokens/Sec on One DeepSeek-V4.1-Flash Node

On the PR's own benchmark against a DeepSeek-V4.1-Flash node, freeing the reserved slots nearly tripled throughput, and every supporting metric moved the same direction at once.

MetricBefore (main)After (PR #59468)
Output throughput (tok/s)1,3733,390
TPOT p50 / p99 (ms)29.2 / 63.515.7 / 36.0
TTFT p50 incl. queueing (s)93.630.0
Acceptance length4.724.73

Acceptance length, how many drafted tokens the target model accepts per verification step on average, stayed flat at roughly 4.72 to 4.73 tokens in both runs. That's the detail that confirms this was a concurrency fix and not a drafting-quality fix: the drafter itself wasn't proposing better tokens, the scheduler was just letting more requests run at once, so queueing time (most of that 93.6-second TTFT) collapsed and per-token latency dropped along with it.

What Else Shipped in the vLLM 0.33 Release Notes

The slot fix is one line in a release built from 927 commits by 389 contributors, including 122 first-time contributors, tagged 1 November 2026 at commit 4ba787e by maintainer khluu. The rest of the release notes worth knowing:

  • CUDA 13.0 only. Release builds dropped CUDA 12.9 wheels and container images; if your deployment image pins 12.9, you'll need to move it before upgrading.
  • Dependency bump. PyTorch moved to 2.14.0, torchvision to 0.29.0, and Triton to 3.8.0.
  • New model support. Reflection Beam-1.0 (a 501B-total, 23B-active MoE model shipping its own DFlash2 drafter), MiniCPM-OCR 1.0, and ColUltraVBERT.

None of those three touch the scheduler. If you're upgrading specifically for the throughput gain, PR #59468 is the one to read.

What Freed Draft Slots Mean for Your GPU Bill

A scheduler bug that silently caps concurrency is a capacity problem wearing a performance costume: the fix doesn't make the GPU faster, it lets the GPU accept more of the work it could already do. For a DFlash or DSpark deployment running near its token-budget ceiling, that means the same node now clears more concurrent requests before you'd otherwise reach for a second GPU or a second node, which is a direct cut to cost per million tokens on exactly the kind of workload these drafters target.

The PR's own hardware line reads "one NVIDIA GB300 node, run with TP4," tensor parallelism split across 4 GPUs, not a single GPU. That's close to a shape Spheron's own fleet has already run: on one 8x B300 spot node (Spheron ES, UK South 1, vLLM 0.33.0, as of 8 Nov 2026), DeepSeek V4.1 Flash was served as two 4-GPU TP copies and reached 6,851.2 tok/s node-wide at 64 concurrent users, a 273ms median time to first token, and $2.01 per million output tokens. That's a different benchmark from the PR's own 1,373-to-3,390 tok/s number, the request mix and prompt lengths differ, but it's the same vLLM version, model, and 4-GPU TP shape, which makes it a second, independent read on what freed-up concurrency is worth per dollar.

Reproducing the PR's exact setup means renting a 4-GPU TP4 slice, not a single on-demand B300 GPU. Spheron sells B300 on-demand and spot in multi-GPU configurations, so that's buildable without a reservation. The full GB300 NVL72 rack, the rack-scale system NVIDIA's naming actually refers to, is a different purchase: it's reservation-only on Spheron with no published hourly rate, the same way it's sales-gated or reservation-only across every major cloud provider that lists it.

Budget for startup time separately from serving cost. On that same 8x B300 spot node, DeepSeek V4.1 Flash took 8 minutes 47 seconds to pull its 510GB of weights and 40 minutes 19 seconds to finish starting, mostly CUDA graph capture and FlashInfer autotuning rather than the download itself, roughly $40.60 of spot time at that run's $49.61/hr average before the first token came back. A scheduler fix that frees draft slots doesn't touch that cost; it's a separate line sitting in front of the gain this post is about.

Pricing fluctuates based on GPU availability. Spheron rates above are live as of 19 Nov 2026; other providers reflect their most recent published rates and may have changed. Check current GPU pricing → for live rates.

A scheduler fix only pays off if the node underneath has spare capacity to absorb the extra concurrency, so test the before/after on the same class of GPU the PR benchmarked before you assume the gain carries over to yours.

Get started on Spheron →

FAQ / 05

Frequently Asked Questions

PR #59468, merged 15 Oct 2026 and shipped in vLLM 0.33.0 on 1 Nov 2026, stops the scheduler from reserving token-budget slots for DFlash and DSpark drafts. Both drafters used to charge the scheduler's token budget as if their draft tokens added rows to the target model's batch, even though neither one does. The charge is now 0, and running requests are capped by max_num_batched_tokens divided by num_lookahead_tokens instead.

No. MTP adds extra prediction heads to the base model itself, trained jointly with it, so a single forward pass proposes several tokens at no separate model cost. Speculative decoding keeps the base (target) model untouched and pairs it with a separate, cheaper drafter, autoregressive, block-based, or diffusion-based, whose proposals the target model then verifies in one pass. DeepSeek's own DSpark method is built to beat its MTP-1 baseline, not replace the concept.

EAGLE-3 drafts one token at a time, autoregressively, using a lightweight head trained on the target model's internal features, with a fixed num_speculative_tokens set once at server launch. DFlash and DSpark both draft a whole block in one shot instead of token by token, which is what made them eligible for the slot-reservation accounting PR #59468 fixes. EAGLE-3's draft head never reserved scheduler slots the same way, so it was never affected by this bug.

No. The PR's own validation ran greedy decoding on 20 fixed prompts with the default scheduling budget, and the outputs were byte-identical between the unpatched build and the patched one. It only changes how the scheduler counts token-budget slots, not what either the drafter or the target model computes.

DFlash on both of vLLM's model runners, and DSpark on the newer MRv2 runner only. Slot accounting for P-EAGLE, PARD, and plain draft-model speculative decoding is unchanged, because those methods were never over-reserving scheduler slots in the first place.

// Try It Yourself

Try It on Real GPUs

The GPUs behind these guides are the ones you can rent here: H100s, H200s, B200s, and more, billed per minute after a 20-minute minimum runtime, with no contracts. Pick one and you are live in under two minutes.

View Pricing
→
Or talk to our team
Spheron GPU instance going live and linking data centers across the globe