SGLang quietly flipped a default in July 2026 that changes how it runs the prefill phase of every request on CUDA. If you run SGLang and haven't touched this flag, your prefill execution changed underneath you the moment you upgraded.
TL;DR: What Does SGLang's Breakable CUDA Graph Default Actually Change?
- Build speed: BCG builds prefill graphs 3.8 to 5.2 times faster than the torch.compile piecewise backend it replaced, per SGLang and Meta's engineering write-up on the v0.5.15 release.
- Replay speed: BCG replays those graphs 17% faster than the same piecewise backend, per the same write-up.
- Default date: SGLang merged PR #29458 making BCG the default prefill backend on CUDA on July 2, 2026, shipping in v0.5.15.
- Real workload: on DeepSeek V4 with DP attention, enabling BCG lifted total throughput 11.80% and cut median time-per-output-token from 230.98ms to 200.32ms in a CoreWeave 8-DP test tied to the same v0.5.15 release.
- Buying consequence: a single-digit-to-teens percentage gain per GPU, worth testing on a per-minute-billed Spheron H100 rental before resizing a fleet around it.
What Breakable CUDA Graphs Fix That Static Graphs Couldn't
A standard CUDA graph captures a fixed sequence of kernel launches once, then replays that exact sequence on every subsequent call, skipping the CPU-side overhead of re-issuing each launch individually. That works well when the computation graph is the same shape every time, which is the entire premise of decode: one token in, one token out, same tensor shapes, every step.
Prefill breaks that assumption. Input sequence length varies with every request, batches are shaped differently every time chunked prefill packs them, and some operations inside prefill, dynamic control flow, host-device synchronization points, JIT compilation calls, simply cannot be captured inside a standard CUDA graph at all. SGLang's own documentation on breakable CUDA graphs puts the two failure modes plainly: when something goes wrong inside a captured graph, "there is no way to step through the operations or insert print statements because the graph replays as a monolithic unit," and separately, certain operations "cannot be captured in standard CUDA graphs" full stop.
Why Prefill Couldn't Use Full CUDA Graph Capture the Way Decode Does
The workaround SGLang shipped before BCG was a torch.compile-based "piecewise" backend: compile the prefill graph with Dynamo, split it at the points that can't be captured, and run captured CUDA graph segments around those breaks. It worked, but it inherited torch.compile's cost structure. Every new input shape, and prefill sees a lot of them, can trigger a fresh compilation pass. That compilation overhead accounted for 78 to 86% of the time the compiler-based backend spent preparing prefill graphs, according to the LMSYS engineering post, with absolute build times reaching 90 seconds on a 235B mixture-of-experts model and 158 seconds on GLM-5.2. For a serving process that needs to warm up quickly or reshape itself as traffic patterns shift, that's a real tax before the first token gets served.
BCG vs the torch.compile-Based Piecewise Backend It Replaces as Default
Breakable CUDA graphs solve the same problem, insert graph breaks so incompatible operations run eagerly between captured segments, without going through a compiler at all. SGLang's documentation frames the fix directly: "Breakable CUDA Graph solves both problems by allowing graph breaks to be inserted at specific points." Because there's no torch.compile pass in the loop, BCG builds prefill graphs 3.8 to 5.2 times faster than the piecewise backend, and it's also 17% faster once both are actually running and replaying captured segments, per the LMSYS blog.
The two approaches also aren't close in implementation size. SGLang and Meta's engineers report reaching the same functional coverage as the torch.compile piecewise backend in 521 lines of code, versus 1,771 lines for the compiler-based approach, a maintenance surface roughly a quarter the size for the same job. If you want the mechanics of the compiler-based side of that comparison, our torch.compile and CUDA graphs guide covers how Dynamo capture and Inductor caching work in production PyTorch 2.6, which is the exact machinery BCG is now defaulting past. The kernel-launch pattern both backends are trying to amortize is also the same one FlashInfer's kernel library is built to reduce further down the stack.
BCG also cuts a real memory cost that comes from capturing at fixed shapes: on gpt-oss-120b, capturing through the chunked-prefill size dropped prefill activation memory peaks from 0.56GB to 0.001GB. On GLM-5.2, the drop was from 1.55GB to 0.35GB, per the same LMSYS write-up. Neither number moves your GPU count on its own, but it's headroom that shows up as fewer OOM retries under bursty prefill load.
The 2026 Default Change: What Shipped and When
The origin point matters here because it's easy to assume this is a PyTorch or CUDA feature that SGLang adopted. It isn't. As SGLang's own engineers, joined by Meta contributors, put it on the LMSYS blog: "Breakable CUDA Graph (BCG) is an SGLang-originated serving technique: it was first proposed, named, implemented, and open-sourced in SGLang."
Timeline: From Prefill Extension to Default
- April 24, 2026: a pull request (PR #22218) merged, extending BCG's graph-break mechanism from decode into the prefill phase for the first time, per SGLang and Meta's engineering write-up on the LMSYS blog. At this point BCG existed for prefill but wasn't the default; teams had to opt in.
- July 2, 2026: PR #29458, titled "Enable Breakable Cuda Graph as Default," merged. This is the flip that matters for anyone running SGLang without touching serving flags: prefill now uses BCG unless one of the fallback conditions below applies.
- v0.5.15: BCG-as-default ships as part of the v0.5.15 release, whose release notes describe the change directly: "Breakable CUDA Graph is now the default capture path, reducing per-step kernel-launch overhead."
What Still Falls Back to Eager Execution
BCG's default applies specifically on CUDA, and only to prefill. Per PR #29458, it auto-disables, meaning prefill runs eager instead, when any of these apply:
| Condition | Why it falls back |
|---|---|
| Context parallelism (attn_cp_size > 1) | Graph capture assumptions break across CP ranks |
| LoRA (lora_paths or enable_lora) | Adapter-dependent computation isn't graph-stable |
| MoE A2A backend enabled | Expert-parallel all-to-all routing varies per step |
| Distributed (DP) attention | Handled separately; see the DeepSeek V4 case below |
| Multimodal models | Image embedding paths aren't covered by the default |
| DeepSeek V4 | Memory pressure considerations, per the PR |
If your deployment runs any of these, the July 2026 default change doesn't touch your prefill path at all, you're still on the prior backend (or eager) until you opt in explicitly or a future release extends coverage. This is also why the DeepSeek V4 DP attention case below shipped as its own PR rather than folding into the default: it needed separate handling, not a flag flip.
The Performance Consequence: Removing Launch Overhead From Dynamic Shapes
BCG's core value proposition is narrow and specific: it removes CPU-side kernel-launch overhead from a phase of inference, prefill, whose input shapes change on every request, without paying a compiler's setup cost to do it. The build-time and replay-time numbers above (3.8 to 5.2x faster builds, 17% faster replay) are gpt-oss-120b and GLM-5.2 figures from SGLang's own benchmarking, not universal constants. Expect the exact percentage to move with your model architecture, sequence length distribution, and batch shape variance; a workload with highly uniform prefill lengths will see less benefit than one with the bursty, mixed-length traffic BCG is designed to help.
Where BCG Actually Touches Decode: DP Attention and Eagle
The prefill default is the headline, but BCG's mechanism has two decode-side applications worth knowing about separately, because they're easy to conflate with the July default and they aren't the same change.
The first is DeepSeek V4 with data-parallel attention. PR #25195 added BCG support for the mixed prefill/extend batches that DP attention produces, which previously had no graph coverage at all and caused what the PR describes as large host-side gaps under high concurrency. In a CoreWeave test on an 8-DP configuration, enabling BCG here lifted total throughput 11.80% (from 8,523.28 to 9,529.16 tokens per GPU per second) and cut median time-per-output-token 13.27%, from 230.98ms to 200.32ms.
The second is Eagle speculative decoding. SGLang enabled breakable CUDA graph support for Eagle in PR #25795, a decode-path use of the same graph-break mechanism, distinct from both the prefill default and the DP attention case. If you're running Eagle already, our Eagle-3 speculative decoding guide covers the draft-head and acceptance-rate tuning that determines how much of Eagle's 3-4x decode speedup you actually see; BCG support for it is an incremental addition on top of that, not a replacement for tuning it correctly. And if your workload is prefill-heavy specifically because you're running long-context or disaggregated setups, the split we cover in prefill-decode disaggregation is the architectural lever that determines how much of your fleet even runs the prefill path BCG is optimizing.
The Cost Consequence: Fewer GPUs for the Same Throughput
Here's what to do with these numbers rather than just admire them. A throughput gain on the same hardware converts directly into fewer GPU-hours for the same request volume, but only for the fraction of your workload where the fallback table above doesn't apply.
Turning an 11-17% Throughput Gain Into GPU-Hours
Take the DeepSeek V4 DP attention result as the clearest worked case, since it's a measured production-shaped benchmark rather than a microbenchmark: 11.80% more total tokens per GPU per second on the same 8-DP CoreWeave configuration. If a fleet needed 10 GPUs to hold a given tokens-per-second target before BCG, an 11.80% throughput lift on that same hardware means the same target is reachable with roughly 8.9 GPUs worth of capacity, call it a 10th GPU you no longer need to provision, assuming your traffic and batch shapes match the benchmark's mixed prefill/extend profile closely enough. That's the honest shape of this optimization: it's a fleet-sizing adjustment, not a hardware-class downgrade.
Worked Example at Current Spheron H100 Pricing
To make that concrete with a number you can actually rent against: Spheron lists on-demand H100 SXM5 80GB at $2.64/hr as of 07 Sep 2026. For the worked math below, we hold the rate fixed at $3.38/hr, the snapshot rate on 6 Sep 2026, so the derived totals stay internally consistent rather than drifting apart from each other; check the live figure above for what it actually costs today. Ten H100s running 24/7 for a month (730 hours) at that frozen rate costs $24,674. Shaving one GPU off that fleet through a BCG-driven throughput gain, holding the same total tokens-per-second target, saves roughly $2,467.40/month, before accounting for the fact that the exact percentage you'll see depends on how close your prefill/decode mix is to the benchmark conditions above. GPU pricing moves with availability and region, so this is illustrative math against a snapshot rate, not a fixed forecast.
Pricing fluctuates based on GPU availability. Spheron rates above are live as of 07 Sep 2026; the worked example uses a frozen snapshot rate of $3.38/hr from 6 Sep 2026, held fixed for consistency. Check current GPU pricing → for live rates.
One caveat worth stating plainly: the official BCG prefill benchmarks in SGLang's own write-up ran on 4xGB300 in a TP4 configuration, and GB300 is listed as "coming soon" on Spheron's pricing page rather than available to rent today. If you want to reproduce those specific numbers, you'll need access to that hardware elsewhere first; the H100 math above is a cost-modeling exercise on hardware you can actually provision now, not a claim that H100 reproduces GB300's exact percentages.
vLLM vs SGLang 2026: Where BCG Shifts the Decision
If you're choosing between vLLM and SGLang for a new deployment in 2026, BCG's default change is a real point in SGLang's favor but not a decisive one. It closes a specific gap, prefill overhead on variable-length, high-concurrency traffic, that previously required opting into a compiler-based workaround with real setup-time cost. It doesn't change the KV-cache reuse story (RadixAttention vs PagedAttention), the structured-output tooling, or the throughput-per-dollar numbers on steady-shape decode workloads, which is most of what our full vLLM vs SGLang 2026 benchmark is actually measuring.
Where BCG's default matters most is exactly the traffic pattern it was built for: agentic and multi-turn workloads with unpredictable input lengths, RAG pipelines with variable retrieved-context sizes, and any deployment running SGLang's production deployment patterns at high concurrency where prefill launch overhead was previously eating into P99 latency. If your workload looks like steady-shape batch decode with little prefill variance, this default change will barely register in your numbers. For the fuller decision framework across vLLM, TensorRT-LLM, and SGLang, including where each one wins on cost rather than just raw throughput, see our LLM inference optimization decision framework.
Mixture-of-experts deployments deserve their own note here: DeepSeek V4 explicitly falls back to eager prefill under the current default, and MoE A2A backend usage does too. If you're running an MoE model on SGLang, check our MoE inference optimization guide before assuming BCG's default is doing anything for your prefill path, because in most MoE configurations right now, it isn't.
Should You Switch From vLLM, or Wait?
If you're already running SGLang and don't hit any of the fallback conditions, there's nothing to decide: upgrading to v0.5.15 or later gets you the default automatically, and the downside case (a workload where BCG genuinely doesn't help) mostly just means you see none of the gain rather than a regression, since the piecewise backend remains available as a fallback path.
If you're running vLLM and evaluating a move, treat this as one input among several rather than the deciding one. An 11-17% throughput gain on the fraction of your traffic that's prefill-bound and doesn't hit a fallback condition is a real number worth having, but it's not the kind of gap that alone justifies a migration if your team already has vLLM tuning, monitoring, and deployment tooling built out. It's a stronger argument if you're starting a new deployment from scratch and haven't committed to either stack yet, or if your existing SGLang deployment has been sitting on the older piecewise backend without anyone reevaluating the flag.
Spheron's documentation on deploying SGLang describes it as agentic LLM serving with RadixAttention for KV cache reuse, constrained decoding, and an OpenAI-compatible API, which is accurate as far as it goes, but it's an overview-level quick guide rather than a page documenting CUDA graph or BCG-specific flags. For the actual SGLANG_USE_BREAKABLE_CUDA_GRAPH environment variable and --debug-cuda-graph flag behavior, SGLang's own breakable CUDA graph documentation is the source to configure against, not a provider's quick-start page.
Testing whether a BCG-default SGLang build actually cuts your fleet's GPU-hours is exactly the kind of thing per-minute billing is for: spin up an H100 on Spheron, benchmark your own prefill/decode mix before committing to a longer rental, and check current numbers instead of trusting a benchmark run on someone else's traffic shape.
Frequently Asked Questions
A breakable CUDA graph (BCG) is a CUDA graph that can be split into multiple captured segments with eager execution allowed in between, instead of forcing the entire forward pass into one monolithic graph. SGLang's own documentation describes it as a way to insert graph breaks at specific points, which lets operations that can't be captured, like dynamic control flow or JIT compilation, run outside the graph while everything else keeps most of the CUDA graph performance benefit.
SGLang merged the change making BCG the default prefill CUDA graph backend on CUDA in pull request #29458 on July 2, 2026, and it shipped as part of the v0.5.15 release. Before that, BCG for prefill existed but had to be turned on explicitly; the earlier PR #22218 that extended BCG to the prefill phase at all merged on April 24, 2026.
No, not by default. PR #29458 changes the default prefill backend only, on CUDA. BCG's decode-side applications, DP attention for DeepSeek V4 and Eagle speculative decoding, shipped as separate pull requests (#25195 and #25795) and are not the same default flip. The prefill default also auto-disables for context parallelism, LoRA, the MoE A2A backend, distributed attention, multimodal models, and DeepSeek V4, all of which fall back to eager prefill instead.
The gains are real but scoped. On the build side, SGLang's engineering blog reports BCG builds prefill graphs 3.8 to 5.2 times faster than the torch.compile-based backend it replaces, and is 17% faster at replay time. On DeepSeek V4 with DP attention specifically, enabling BCG lifted total throughput 11.80% and cut median time-per-output-token from 230.98ms to 200.32ms in a CoreWeave 8-DP test. That is a single-digit-to-teens percentage gain per GPU, not a step-change in tokens per second.
