We went into this with a specific hypothesis: acceptance rate, the number that decides whether speculative decoding actually pays for itself, moves a lot more by task type than the single headline speedup most writeups quote. Speculative decoding explained in one line is easy: a small draft model guesses several tokens ahead, and the target model you actually care about checks all of them in one pass instead of grinding through them one at a time. What's harder to find is whether that guess holds up the same way for a support chatbot as it does for a code assistant, and that's the question we set out to test.
When it works, a draft model turns one token per expensive forward pass into two to four. When it doesn't, you're running two models and getting close to the throughput of one. Acceptance rate is the number that tells you which situation you're in, and it's rarely broken out by task in the material written about this technique. So we lined up acceptance-rate results across task types against a dense-decoding baseline, and checked our hypothesis against what three separate research teams had already measured. What surprised us is below, along with the number that actually decides whether this is worth configuring on the GPU tier you're already renting, or whether you should save the engineering time and pay for a bigger one instead.
The Decode Bottleneck: One Token at a Time
Standard autoregressive decoding generates exactly one token per forward pass. To generate that token, the GPU has to read the entire model's weights (and the growing KV cache) out of HBM, run the math, and write one token back. Then it does the whole thing again for the next token.
At batch size 1, that forward pass is almost entirely memory movement, not arithmetic. The GPU's compute units finish the matrix multiplications faster than the memory bus can deliver the next layer's weights, so they sit idle waiting on data. This is the same memory-bandwidth wall that shows up in FlashAttention's tiling problem: the fix there is moving less data per attention step, and the fix in decode is generating more tokens per trip through the weights.
That's the part a bigger GPU doesn't structurally solve. Moving from an H100 to an H200 buys you more HBM bandwidth, which shortens each forward pass a little, but you're still emitting one token per pass. You've made the walk faster; you haven't made it shorter. A draft model changes the shape of the walk itself.
How Speculative Decoding Buys You Multiple Tokens Per Step
The technique was formalized in "Fast Inference from Transformers via Speculative Decoding": run a small, cheap draft model autoregressively for a few steps to propose a short sequence of candidate tokens, then have the large target model verify all of them in a single forward pass. Tokens the target model agrees with are kept; the first one it disagrees with is resampled from the corrected distribution, and everything after that is discarded and redone. The math behind the rejection sampling step guarantees the output distribution is identical to running the target model alone, token for token. You're not trading quality for speed here, you're trading idle compute for speed.
A parallel paper from DeepMind, "Accelerating Large Language Model Decoding with Speculative Sampling" (Chen et al., 2023), demonstrated the same idea on Chinchilla 70B and reported a 2-2.5x decoding speedup in a distributed setup. Both papers land on the same conclusion from different labs: verification is parallel, drafting is cheap, and the combination turns a bandwidth-bound loop into a batched check.
The obvious follow-up question is how good the draft model has to be, and that's where the EAGLE family changed the design. EAGLE-3 pushed this further with training-time test scaling and reports a speedup ratio up to 6.5x over standard decoding, about 1.4x faster than EAGLE-2, plus a 1.38x throughput improvement at batch size 64 in SGLang. We cover the exact checkpoints and config flags for running it in our EAGLE-3 speculative decoding deployment guide.
The newest wrinkle is P-EAGLE, which fixes a structural inefficiency in EAGLE itself: as the vLLM blog post announcing it explains, "EAGLE drafts tokens autoregressively. To produce K draft tokens, it requires K forward passes through the draft model." P-EAGLE generates all K draft tokens in a single forward pass instead, and the team reports up to 1.69x speedup over vanilla EAGLE-3 on an NVIDIA B200.
Worth flagging one adjacent approach that solves the same one-token-per-step problem differently: multi-token prediction bakes extra prediction heads directly into the target model's own weights during training, so there's no separate draft model to load or verify against at all. It's a different tradeoff, no second model in VRAM, but it only works on models trained with MTP heads from the start. We break down when that's the better fit in our multi-token prediction deployment guide.
Acceptance Rate: The Number That Decides If Speculative Decoding Pays Off
Acceptance rate is the fraction of a draft model's proposed tokens the target model confirms instead of rejecting on verification. As the Snowflake engineering team behind Arctic Inference put it, "a high acceptance rate signifies that the draft model is adept at predicting the output of the target model." Every other number in this technique, the 2-6.5x speedups you'll see quoted, is downstream of this one. A draft model that guesses right most of the time turns a handful of cheap autoregressive steps into several free tokens per verification pass. A draft model that guesses wrong constantly means you paid for its forward passes and got almost nothing back beyond standard decoding speed.
This is also why speculative decoding isn't a free multiplier you flip on everywhere. At high batch sizes, the verification pass itself gets more expensive because it's checking candidate tokens across every concurrent sequence in the batch, which erodes the advantage; our production guide's rule of thumb is to avoid it above roughly 32 concurrent requests. And acceptance rate itself is not fixed. It depends on how well the draft model was trained to match the target model's specific behavior on your traffic, not just its architecture. Snowflake reports that its Arctic Inference speculators, trained on production traffic rather than a generic public dataset, reach 3.1x higher acceptance rates than public baseline drafters on ShareGPT, which is a bigger swing than most people expect from "just" retraining the draft model on your own data.
Our Hypothesis and Test Setup
Our hypothesis going in: acceptance rate isn't one number, it's a property of the pairing between draft model, target model, and task. A draft model that predicts chat completions well should predict code completions differently, because the token distributions look nothing alike, one is closer to natural language, the other is closer to a formal grammar. If that's true, a single blended acceptance-rate figure for "speculative decoding" is close to useless for deciding whether it's worth configuring on your own workload.
We don't have three independently trained draft models to A/B against the same target model in a single sitting, so the fairest way to test that hypothesis right now is to line up what three separate research teams already measured, each holding task type roughly constant while varying the drafting method: Leviathan et al.'s T5-XXL results, Chen et al.'s Chinchilla 70B results split between general decoding and HumanEval, and the P-EAGLE team's per-task breakdown from vLLM. All three verify against a standard dense-model baseline, the same target model running plain autoregressive decoding with no draft model at all, which is the comparison point that matters for the buying question.
What We Measured Against a Dense Baseline
Acceptance-Rate Table by Task Type
| Task type | Draft approach | Result vs. dense baseline | Source |
|---|---|---|---|
| General text (T5-XXL) | Original speculative decoding | 2x-3x acceleration versus standard T5X decoding, with identical outputs | Leviathan et al., 2023 |
| General decoding (Chinchilla 70B) | Speculative sampling | 2-2.5x decoding speedup on Chinchilla 70B in a distributed setup | Chen et al., DeepMind, 2023 |
| Code generation (HumanEval, Chinchilla 70B) | Speculative sampling | rising to almost 2.5x on HumanEval, without any change to output quality | Chen et al., DeepMind, 2023 |
| Code generation (HumanEval, K=7 depth) | P-EAGLE vs. EAGLE-3 | P-EAGLE outperforms EAGLE-3 by 30% on HumanEval (3.94 vs 3.03 average accepted tokens) | vLLM blog, P-EAGLE team |
| General decoding | EAGLE-3 vs. standard decoding | speedup ratio up to 6.5x, about 1.4x over EAGLE-2 | EAGLE-3 paper |
| Chat (ShareGPT) | Arctic Inference speculators vs. public baseline drafters | 3.1x higher acceptance rates than public baseline drafters | Snowflake Engineering |
Pricing fluctuates based on GPU availability. The prices referenced later in this post are based on 3 Sep 2026 and may have changed. Check current GPU pricing → for live rates.
What Surprised Us
The gap our hypothesis predicted showed up, but not in the direction we expected. We assumed code generation would have the lowest acceptance rate, on the theory that code's syntax is more brittle and a wrong token derails a whole line. The opposite showed up in the P-EAGLE numbers: code (HumanEval) is where the newer drafting method pulled furthest ahead, with a 30% jump in average accepted tokens over EAGLE-3 at K=7. Code's rigid grammar is apparently easier for a well-trained draft model to predict correctly, not harder, once the drafting method itself is good enough to exploit that structure.
The second surprise was about speculation depth. It's tempting to assume that letting the draft model propose more tokens ahead (a higher K) always helps, since more guesses means more chances to save a verification pass. The P-EAGLE data says otherwise: P-EAGLE's peak throughput lands at K=7, while vanilla EAGLE-3 peaks earlier, at K=3, and degrades past that point. Going deeper only pays off if the drafting method is fast enough per step to make the extra guesses cheap; bolt a high K onto a slower drafting method and you're burning time on speculative tokens that were never going to get accepted anyway. K is not a dial you can crank up blindly, it has to be tuned against the specific draft model you're running.
Buying Takeaway: Hitting a Latency SLA Without a Bigger GPU
If your service is breaching a TTFT or inter-token latency SLA, the default move is to rent a GPU with more memory bandwidth and hope the shorter forward pass closes the gap. That works, but it's usually the expensive fix for a problem a draft model solves more directly. As of 3 Sep 2026, an H100 on Spheron runs $2.65/hr on-demand ($2.20/hr on spot); stepping up to an H200 for its extra bandwidth nearly doubles that to $5.28/hr on-demand, a cost you're paying every hour the instance is up, whether or not the extra bandwidth is what your workload actually needed.
A draft model attacks the same bottleneck for a fraction of that cost. The VRAM overhead of a 1-3B draft model, or an EAGLE-style single-layer draft head, is small next to a full tier upgrade, but you do need to confirm your current GPU has the headroom to hold it alongside the target model; our GPU memory requirements guide walks through that math before you commit to a config. And the acceptance-rate table above is the reason this isn't a blind bet: if your workload looks like chat or code, the published numbers say a decent draft model should buy real headroom back; if it looks like open-ended long-form generation with unpredictable next tokens, the gain will be thinner, and it's worth checking before you invest engineering time in the config.
Spheron's per-minute billing and no minimum rental period make that check cheap to run. You can spin up the same GPU tier you're already on, wire in a draft model, and measure your own acceptance rate on your own traffic in an hour of instance time rather than committing to a bigger machine on a guess. Spheron aggregates on-demand and spot pricing across 5+ providers, so if the draft model genuinely isn't enough and you do need more bandwidth, the next tier up, H200, B200, B300, is available on the same platform without switching providers. To be clear about the split: Spheron's pricing page covers the hardware and the billing, not the speculative decoding config itself; for the actual vLLM and SGLang flags, draft-model checkpoints, and per-GPU cost-per-token numbers, that's what our production guide is for.
Testing whether a draft model closes your latency gap costs an hour of instance time, not a tier upgrade you're stuck paying for either way.
Frequently Asked Questions
Speculative decoding is an inference technique where a small, fast draft model proposes several candidate tokens ahead of the target model, and the target model checks all of them in one forward pass instead of generating them one at a time. Accepted tokens come out for free; rejected ones fall back to standard decoding. The output distribution is identical to running the target model alone, so nothing about quality changes, only how many tokens you get per expensive pass.
Acceptance rate is the share of a draft model's proposed tokens that the target model confirms rather than rejects during verification. It is the single number that decides whether speculative decoding is worth running on a given workload: a high acceptance rate means most speculative work turns into real output, while a low one means you are paying for two forward passes and getting close to the throughput of one.
No. It helps most in low-concurrency, latency-sensitive serving, where decode is memory-bandwidth-bound and idle compute exists to spend on drafting. At high batch sizes the verification pass itself gets expensive and the advantage shrinks, and on tasks where the draft model can't predict the target model's next few tokens well, low acceptance eats the gain entirely. It's a workload-dependent optimization, not a universal multiplier.
Often, yes. Decode latency on a single GPU is set mostly by memory bandwidth, and a draft model attacks that bottleneck directly by turning one token per pass into several. If your P99 time-to-first-token or inter-token latency is missing budget on your current GPU tier, adding a compatible draft model is worth testing before paying for a higher-bandwidth GPU, since the VRAM and dollar cost of a draft model is a fraction of what a tier upgrade costs.
