SPEED-Bench is a speculative decoding benchmark built around one uncomfortable question: what happens when you stop feeding a draft model random tokens and start feeding it the prompts your users actually send? The answer is a 23% gap between what the old-style benchmarks reported and what SPEED-Bench measures running inside a real inference engine. That gap isn't a rounding error in a footnote. It's large enough to change how many GPUs a spec-decoding-enabled fleet actually needs.
This matters beyond one paper. Every speculative decoding speedup number circulating right now, including the ones on this site, was measured under specific conditions that don't automatically transfer to your traffic mix. We cover the mechanics of draft-verify decoding in our speculative decoding production guide and walk through the acceptance-rate concept itself in speculative decoding explained. This post is about what SPEED-Bench adds on top of that: the first production-engine benchmark to actually quantify how much a synthetic test overstates reality, and how far acceptance rate swings once you stop averaging across task types.
TL;DR: Does SPEED-Bench Prove Speculative Decoding Benchmarks Overstate Real Throughput?
- Overstatement: NVIDIA's SPEED-Bench found that synthetic, random-token testing overstates real-world speculative decoding throughput by roughly 23% compared to its own real-prompt measurement.
- Domain spread: SPEED-Bench's Qualitative split tests 880 real prompts across 11 categories, from Coding and Math to Roleplay and Writing, because the paper's authors found SD performance "inherently data-dependent."
- Bimodal, not average: In NVIDIA's own 2026 AIPerf reference run, 13% of verify steps accepted zero draft tokens while 61% accepted the full 5-token draft, a spread that one average acceptance figure hides completely.
- Spheron bills H100 SXM5 at $2.39/hr on-demand as of 22 Sep 2026, cheap enough to run your own acceptance-rate check on your real traffic before sizing a fleet around someone else's demo. Read the speculative decoding production guide for the flags this depends on.
What NVIDIA's SPEED-Bench Found: Random Tokens vs Real Prompts
Older speculative decoding benchmarks had a structural problem, and SPEED-Bench's own paper names it directly in its abstract: existing benchmarks "suffer from limited task diversity, inadequate support for throughput-oriented evaluation, and a reliance on high-level implementations that fail to reflect production environments." Translate that out of paper-speak and it says three things. Test prompts didn't look like real traffic. Latency got measured instead of sustained throughput. And the harnesses running the tests weren't the same code paths a production server actually executes.
That third point is the one SPEED-Bench fixes first. SPEED-Bench integrates directly with vLLM, TensorRT-LLM, and SGLang rather than a standalone research script, specifically so its numbers reflect what a deployed server would see, not what an isolated draft-verify loop produces in a notebook. That single design choice is why the 23% gap is worth taking seriously: it isn't measuring a different metric, it's measuring the same metric more honestly.
The random-token problem is more specific than "unrealistic." NVIDIA's researchers point out that random-token inputs, the input type most older SD benchmarks defaulted to, fail to trigger realistic expert routing in MoE models, leaving a meaningful share of experts in some layers never activated during the benchmark run. If your target model is a mixture-of-experts architecture (and increasingly it is, given how much of the frontier open-weight lineup has moved that direction), a random-token benchmark isn't just noisy, it's exercising a different model than the one your users hit. We go deeper on why MoE routing complicates draft prediction specifically in our speculative decoding for MoE models guide, and the SPEED-Bench routing finding is the external benchmark confirmation of exactly that mechanism.
To fix the diversity problem, SPEED-Bench's Qualitative split runs 880 prompts across 11 categories, Coding, Math, Humanities, STEM, Writing, Summarization, Roleplay, RAG, Multilingual, Reasoning, and QA, drawn from roughly two dozen source datasets. And instead of testing one speculative decoding method on one model pair, the paper evaluated N-Gram, vanilla SD, EAGLE-3, and native multi-token-prediction heads across Llama 3.3 70B, GPT-OSS 120B, DeepSeek-R1, Qwen3 235B, and Qwen3-Next, specifically to surface how much acceptance rate depends on the model and domain rather than the SD method alone.
Put the design and the finding together and you get the headline number: synthetic, random-token benchmarking overestimates real-world speculative decoding throughput by approximately 23% compared to what SPEED-Bench measures with real prompts running inside a production engine. That's the gap between a vendor slide and a production dashboard, quantified for the first time by the people who make the GPUs those numbers get measured on.
vLLM Speculative Decoding in Production: Why Acceptance Rate Swings by Domain
Acceptance rate is the single number that decides whether speculative decoding is worth running at all, and it's not a fixed property of a draft model and target model pair. It's a property of what you're asking the pair to generate. Talor Abramovich and the other NVIDIA researchers behind SPEED-Bench state this plainly in the paper: "SD performance is inherently data-dependent, meaning that diverse and representative workloads are essential for accurately measuring its effectiveness." That's not a caveat buried in a limitations section. It's the paper's central justification for building an 11-category benchmark instead of a single-domain one.
The mechanism behind that data-dependency is straightforward once you think about what a draft model is actually predicting. On a constrained task like code completion or a math proof, the next few tokens are often heavily determined by syntax and convention: a closing parenthesis follows an opening one, a variable name repeats, a proof step follows a known pattern. A lightweight draft model trained on general text has a real shot at guessing those tokens correctly. On an open-ended task like creative writing or roleplay dialogue, the next token is one of many plausible continuations, and a small draft model has far less signal to work with. SPEED-Bench's 11-category taxonomy exists precisely to stop a benchmark from averaging away that difference between Coding and Math on one end and Roleplay and Writing on the other.
You don't have to take that mechanism on faith. NVIDIA's own AIPerf tutorial for running SPEED-Bench shows the accepted-draft distribution is bimodal, not a smooth curve: in a reference run, 13% of verify steps accepted zero draft tokens while 61% accepted the full 5-token draft. Average those two numbers together and you get a tidy acceptance rate that describes neither regime accurately. A fleet sized off that average is either overprovisioned for the easy traffic or underprovisioned for the hard traffic, and most production systems carry both in the same request queue.
Here's how that shows up in vLLM specifically. The engine already exposes the raw counters you need, you just have to stop reading them as one fleet-wide number:
# Fleet-wide average (hides the bimodal split)
rate(vllm:spec_decode_num_accepted_tokens[5m]) / rate(vllm:spec_decode_num_draft_tokens[5m])
# Same query, scoped to a request label you attach at the proxy layer
rate(vllm:spec_decode_num_accepted_tokens{route="code-assistant"}[5m])
/ rate(vllm:spec_decode_num_draft_tokens{route="code-assistant"}[5m])
rate(vllm:spec_decode_num_accepted_tokens{route="support-chat"}[5m])
/ rate(vllm:spec_decode_num_draft_tokens{route="support-chat"}[5m])That's a config change, not a rewrite: tag requests at whatever layer already knows the traffic type (an API route, a system prompt template, an upstream service name) and let Prometheus split the acceptance-rate query on that label. Our speculative decoding production guide covers the full --speculative-model and --num-speculative-tokens flag set this metric depends on; the label-splitting step above is what turns a single fleet average into the per-domain picture SPEED-Bench says you actually need.
The Gap Between Demo Numbers and Your Actual Traffic Mix
The speedup numbers vendors quote, including our own, are usually measured on a specific task type, and the fine print matters more than the headline multiplier. Our EAGLE-3 deployment guide reports 3-4x throughput gains with acceptance rates of 0.80-0.88, and it says so explicitly: those numbers hold on coding and instruction-following tasks. Our DFlash guide reports a 6x lossless speedup, again measured on instruction-following workloads. Neither post is wrong. Both are describing real results on a real, narrower slice of traffic than "speculative decoding" as a category.
The trouble starts when that domain-specific number gets read as a fleet-wide constant. Production traffic is rarely one domain. A coding assistant fields plenty of plain-English clarifying questions between code blocks. A customer support bot occasionally gets asked to draft something more open-ended. A RAG pipeline mixes structured lookups with free-form summarization. SPEED-Bench's contribution here isn't a new speedup number to replace the old ones, it's proof that averaging across that mix understates how much the acceptance rate, and therefore the throughput gain, moves depending on which slice of the mix is actually running at any given moment.
This is the same pattern we've traced in two other speculative-decoding-adjacent myths on this blog. Quantization doesn't reliably speed up inference either; it depends on whether the workload is memory-bound or compute-bound at the batch size you're actually running. Doubling batch size doesn't double throughput once KV cache pressure or the compute ceiling gets in the way. SPEED-Bench is the same shape of correction applied to speculative decoding: the headline multiplier is real, but it was measured under conditions that a synthetic benchmark, or a single-domain production benchmark, doesn't disclose loudly enough.
What This Means for Sizing Your Spec-Decoding-Enabled Fleet
The buying consequence is direct: size a spec-decoding fleet off your own measured acceptance rate on your own traffic mix, not off a vendor's single-domain demo number. A 23% gap between synthetic and real-prompt throughput, layered on top of a bimodal acceptance distribution that swings between 0% and 100% draft acceptance within the same reference run, is enough to turn "we need 4 GPUs" into "we need 5" or push a marginal spec-decoding deployment back to standard decoding entirely for the slice of traffic where it isn't earning its keep.
The practical version of that exercise is cheap to run. Provision a GPU, enable speculative decoding with the flags from the production guide, route a sample of your real traffic through it, and read the per-domain acceptance rate off the /metrics endpoint using the labeled query above. If coding and structured-QA traffic clears 0.75+ acceptance while your open-ended traffic sits closer to 0.4, that's your real fleet math, not the number from a benchmark slide.
This is one of the places bare-metal access genuinely matters more than an API call. Swapping a draft model, changing --num-speculative-tokens, or comparing an EAGLE-3 checkpoint against a vanilla draft model to see how acceptance rate actually moves on your prompts needs direct access to the serving stack, not a managed inference endpoint that's already made those choices for you. Spheron delivers instances as a fully provisioned VM or bare metal instance with a dedicated IP address and full root access, and bills per minute with a 20-minute minimum runtime and no long-term contract, so a same-day acceptance-rate test costs a few dollars instead of a procurement cycle. That matters because Spheron doesn't run the benchmark or choose the draft model, acceptance threshold, or GPU count for you; your real acceptance rate is a property of your own prompt distribution, and no GPU provider can hand you that number without you running the test on your own traffic. A team that wants a fully managed inference platform with acceptance-rate tuning already built in is better served by a managed API than by renting raw compute to measure it yourself. If you're renting the compute to run that measurement, Spheron's docs list the available GPU instance types and how quickly a deployment spins up.
Once you have that number, the fleet-sizing question stops being a guess. Multiply your measured tokens/sec by domain against your actual traffic split by domain, not against the single number on whichever deployment guide you read first, including ours.
Pricing fluctuates based on GPU availability. Spheron rates above are live as of 22 Sep 2026; check current GPU pricing for live rates.
The gap between a benchmark slide and your production dashboard is exactly what SPEED-Bench set out to measure, and now you can measure your own.
Frequently Asked Questions
SPEED-Bench's paper evaluates N-Gram, vanilla speculative decoding, EAGLE-3, and native multi-token-prediction heads across five target models, Llama 3.3 70B, GPT-OSS 120B, DeepSeek-R1, Qwen3 235B, and Qwen3-Next, rather than a single model pair. NVIDIA's researchers built it this way specifically to test whether acceptance rate depends more on the SD method or on the model and domain being served.
Usually yes, but the fleet math has to come from your own measured acceptance rate rather than a vendor's demo number. A 23% overstatement doesn't mean speculative decoding stops helping, it means the multiplier you plan a GPU count around should come from your own traffic, not from a benchmark run on random tokens or measured on a single narrow domain.
Query vLLM's /metrics endpoint for vllm:spec_decode_draft_acceptance_rate (or derive it via rate(vllm:spec_decode_num_accepted_tokens[1m]) / rate(vllm:spec_decode_num_draft_tokens[1m])) and split the query by request labels for your actual traffic segments instead of reading one fleet-wide average. Spheron bills GPU instances by the minute with a 20-minute minimum and no long-term contract, so running that check on your own prompt mix costs a few dollars, not a procurement cycle.
It means one average acceptance figure hides two different operating regimes instead of describing a typical one. Size a fleet off that average and you risk being overprovisioned for the easy, high-acceptance traffic and underprovisioned for the hard, low-acceptance traffic sitting in the same request queue. Splitting the acceptance-rate query by traffic-type label, the same way you'd check per-domain throughput, shows which regime your queue is actually running before you commit to a GPU count.
