Kernel fusion is the reason a serving stack running torch.compile or a hand-tuned Triton kernel needs measurably fewer GPU-hours to serve the same requests, and it works through a mechanism you can check with a stopwatch: it removes trips through HBM and it removes kernel launches, and it does so at the exact spots in a transformer where those two costs dominate.
TL;DR: What Is Kernel Fusion?
Kernel fusion combines multiple GPU operations into one kernel launch, so intermediate results stay in registers instead of round-tripping through HBM.
- Mechanism: a fused kernel keeps intermediates on-chip instead of writing each op's output to HBM and reading it back.
- Measured win: NVIDIA's
sum(abs(x))fusion on an RTX 4090 cut runtime from 3.51ms to 1.18ms and HBM traffic from 3GB to 1GB. - Launch tax: CUDA API overhead runs about 10 microseconds per unfused kernel launch, per Modal's GPU glossary.
- Where it fails: compute-bound matmuls and atomic-heavy kernels see little or no gain from fusion.
- Spheron bills H100 and A100 GPUs per minute, so GPU-hours a fused build saves are hours not billed.
Why Every Unfused Op Round-Trips to HBM
A GPU's compute cores (the streaming multiprocessors, or SMs) are fast. Its device memory, HBM, is fast too by CPU standards, but it is not nearly as fast as the SMs sitting next to it. NVIDIA's technical blog on the subject puts the problem plainly: "A common bottleneck when writing GPU code is that GPU compute is so fast that even high-bandwidth device memory doesn't use the GPU kernel fully. Kernel fusion addresses this by combining multiple GPU operations into a single device kernel, so intermediate results don't need to round-trip through global memory or require separate kernel launches."
Every operation in an unfused chain, an elementwise add, a ReLU, a LayerNorm, a softmax, follows the same pattern: load its inputs from HBM into on-chip registers, compute, then write the result back out to HBM so the next kernel can read it. If you chain five operations, you get five separate read-then-write round trips through the slowest part of the pipeline, even though the intermediate values never needed to leave the chip in the first place.
The Roofline Test: Why Elementwise Ops Are Memory-Bound, Not Compute-Bound
The roofline model is the standard way to classify a kernel: plot its arithmetic intensity (floating-point operations per byte moved) against the GPU's peak compute and peak memory bandwidth, and see which one the kernel actually hits first. A kernel is memory-bound when it moves more bytes per useful computation than the hardware's compute-to-bandwidth ratio supports, meaning the SMs finish their math and then sit idle waiting on HBM.
Elementwise ops (add, multiply, activation functions like GELU and SiLU, normalization layers) are the textbook memory-bound case. Each output element requires only a handful of floating-point operations, but every element still has to be read from and written to HBM. Fusing several of these ops together doesn't add meaningful compute time, since the SMs were never the bottleneck. It removes memory traffic, which is the part of the pipeline that was actually limiting the kernel.
The worked example on NVIDIA's technical blog measures this directly. An unfused two-kernel sum(abs(x)) reduction on an RTX 4090 ran in 3.51ms total (2.28ms for the first kernel, 1.23ms for the second) while moving 3GB through HBM at roughly 855 GiB/s effective bandwidth. Fusing the two kernels into one cut runtime to 1.18ms and HBM traffic to 1GB, close to a 3x speedup. Bandwidth utilization barely moved, from about 855 to 851 GiB/s, because the fused kernel wasn't using memory more efficiently per byte. It simply had fewer bytes to move.
Kernel Launch Overhead: The Second Tax on Every Unfused Op
Memory traffic isn't the only cost of splitting work across multiple kernels. Every kernel launch also carries fixed overhead: the CPU has to issue a CUDA API call, the driver has to queue the work, and the GPU has to schedule it onto available SMs before any actual computation starts. Modal's GPU glossary puts CUDA API call overhead on the order of 10 microseconds per kernel launch, a tax that applies before any compute runs regardless of how small the kernel's actual work turns out to be.
Ten microseconds sounds trivial next to a multi-millisecond forward pass, until you count how many kernels a single transformer layer launches. A layer with separate kernels for QKV projection, RoPE, attention, output projection, residual add, LayerNorm, and the MLP's activation and second matmul can easily hit a dozen or more launches, and a full model has dozens of layers. Chain enough of those together and the pure launch tax, before a single FLOP has run, adds up to hundreds of microseconds to milliseconds per forward pass. On a serving workload processing thousands of requests, that tax gets paid on every single one.
Kernel Fusion in Practice: torch.compile, Triton, and Hand-Written Kernels
Fusion isn't one technique, it's a category of compiler and hand-written optimizations that all chase the same goal: fewer HBM round trips, fewer launches. Three mechanisms cover most of what a team running inference on rented GPUs will actually touch.
Vertical Fusion: How torch.compile's Inductor Backend Fuses by Default
torch.compile()'s default backend, TorchInductor, traces the operations in a model's forward pass into a graph and applies fusion passes automatically, no custom kernel code required. The most common pattern it applies is vertical fusion: chaining pointwise and reduction operations that sit next to each other in the graph, like a bias add followed by an activation followed by a dropout mask, into a single generated kernel. This is the same class of fusion NVIDIA's sum(abs(x)) example demonstrates by hand, just applied automatically across an entire model graph instead of one operation pair.
The practical upshot is that a team doesn't need to write Triton or CUDA to capture a meaningful share of the fusion win. Wrapping a model in torch.compile() and letting Inductor's fusion passes run over the elementwise and normalization chains in a transformer recovers a real portion of the memory-bound savings automatically, which is why it's usually the first thing worth trying before reaching for hand-written kernels.
GEMM Epilogue Fusion and Horizontal Fusion: Two Other Patterns
Vertical fusion isn't the only shape this takes. GEMM epilogue fusion folds the operations that immediately follow a matrix multiplication, a bias add, an activation, a residual connection, directly into the same kernel that computes the matmul, so the GEMM's output tile never has to round-trip to HBM before the epilogue runs on it. This is a common target for hand-written kernels and libraries like CUTLASS, since the matmul itself is compute-bound but its epilogue is memory-bound, and fusing the two lets the epilogue ride along essentially free.
Horizontal fusion combines independent operations that don't depend on each other's output but share an input, common in multi-head attention where several projection matmuls read from the same input tensor. Instead of launching separate kernels that each re-read that shared input from HBM, a horizontally fused kernel loads it once and produces multiple outputs from a single pass.
Where Hand-Written Triton and CUTLASS Kernels Still Beat Auto-Fusion
Compiler-driven fusion has limits. TorchInductor's default fusion heuristics work well on the common vertical and simple horizontal patterns, but they don't always find the optimal fusion boundary on unusual memory access patterns, ragged batch structures, or ops with data-dependent control flow, the kind of case where a human who understands the exact memory layout can do meaningfully better. FlashAttention is the extreme version of this: it fuses an entire attention block, the QK matmul, softmax, and the output matmul, into one kernel so the N x N attention score matrix, which for long sequences is far larger than the input itself, never touches HBM at all. Our FlashAttention explainer covers the tiling math behind that fusion in detail.
This is also where a library like Liger Kernel earns its place in a training stack: it's a set of hand-written fused Triton kernels that patch directly into the HuggingFace Trainer, replacing the eager-mode PyTorch implementations of RMSNorm, RoPE, and cross-entropy loss with fused equivalents that a generic compiler pass doesn't automatically discover. On the inference side, FlashInfer ships fused kernels purpose-built for serving, including fused RoPE and fused sampling, that vLLM and SGLang call directly rather than relying on torch.compile to find the same fusion at runtime. If you want to write this class of kernel yourself, our Triton kernel development guide walks through authoring and benchmarking one in Python without CUDA C++.
Fusion isn't the only lever for closing the gap between a kernel's theoretical and achieved performance, either. Once a kernel is already fused, techniques like warp specialization and the Tensor Memory Accelerator overlap the loads that remain with compute, which our producer-consumer kernel pattern piece covers as the complementary technique: fusion cuts how much memory traffic exists in the first place, overlap hides the latency of whatever traffic is left.
What Fusion Is Worth in GPU-Hours on a Real Serving Workload
The number that matters to a buyer isn't milliseconds, it's whether a fused build lets the same GPU fleet clear the same request volume in fewer billed hours, or whether it lets a smaller fleet clear the same volume at all.
The Measured Range: 1.5x-3.13x on Memory-Bound Ops
The NVIDIA sum(abs(x)) example is a single, clean data point: a roughly 3x speedup (3.51ms to 1.18ms) from fusing two memory-bound kernels on an RTX 4090, driven entirely by the 3GB-to-1GB drop in HBM traffic, since bandwidth utilization itself barely changed. That single measured result sits inside the range this piece's research draws on for memory-bound fusion generally, 1.5x to 3.13x, which is the shape to expect: real, multiplicative, and dependent on how much HBM traffic the unfused version was actually generating relative to its compute.
The mechanism generalizes past this one example. Any chain of elementwise or reduction ops, activation functions, normalization layers, residual adds, that currently runs as separate kernels is paying the same HBM round-trip tax NVIDIA's example measures directly. The exact multiplier moves with how many ops are in the chain and how much data each one touches, but the direction and the order of magnitude hold.
When Fusion Doesn't Help: Compute-Bound and Atomic-Heavy Kernels
Fusion is not a universal multiplier, and treating it as one is how a benchmark headline turns into a bad buying decision. A large matrix multiplication, the kind that dominates a transformer's linear layers, is compute-bound on modern tensor cores: the SMs are already the bottleneck, not HBM bandwidth, so fusing a GEMM with more surrounding memory traffic doesn't remove a constraint that wasn't binding in the first place. Epilogue fusion still helps here, but only for the memory-bound epilogue riding alongside the matmul, not the matmul itself.
Kernels leaning on atomic operations are a second case worth flagging separately. Atomics serialize writes to shared memory locations, and cramming more work into a single fused kernel with heavy atomic contention can lengthen that serialized section rather than shrink it, eating into or erasing the launch-overhead savings. This is also where fusion runs into the register-pressure tradeoff our CUDA occupancy piece covers: a fused kernel that combines more operations into one kernel body typically needs more live registers per thread, which can lower occupancy and, on a kernel that was relying on many resident warps to hide latency, cost back some of what fusion just saved. The fix is rarely "don't fuse," it's checking achieved occupancy after fusing rather than assuming the fused kernel is strictly better because it launches fewer times.
Fusion's payoff also isn't hardware-portable by default. The same fused Triton kernel source can behave differently across CUDA and ROCm, which our Triton on AMD ROCm vs NVIDIA CUDA piece traces directly: fusion strategy is a real, hardware-dependent variable, not a constant you can assume transfers unchanged between GPU vendors or architectures.
Turning Saved Milliseconds Into Fewer Billed GPU-Hours
The conversion from milliseconds to dollars runs through the same formula our cost per million tokens breakdown uses: cost per token is GPU price divided by measured tokens per second, not the sticker rate. Fusion's job in that formula is entirely on the denominator, it raises tokens per second by cutting the per-request latency spent moving data through HBM and launching kernels, without touching the GPU's hourly price. A serving stack that clears requests faster clears more of them inside the same billed hour, which is the only lever that matters once the hourly rate is fixed.
That's also the practical argument for renting rather than reserving while you validate a fusion change. Spheron's GPU marketplace bills per minute after a 20-minute minimum runtime, with no long-term contract. A torch.compile or FlashInfer build that cuts real GPU-hours shows up directly as fewer minutes billed, not as an unused block on a reservation a team is stuck paying for regardless of whether the fused kernel actually helped.
Pricing fluctuates based on GPU availability. Spheron rates are live as of 24 Sep 2026. Check current GPU pricing → for live rates.
A torch.compile Toggle Test You Can Run Yourself
You don't need NVIDIA's benchmark rig to check whether fusion is doing anything for your own model. It's the same experiment twice, minutes apart, on one rented GPU:
import time
import torch
model = MyModel().cuda().eval()
x = torch.randn(BATCH, SEQ_LEN, HIDDEN, device="cuda")
def bench(fn, iters=50):
torch.cuda.synchronize()
start = time.perf_counter()
for _ in range(iters):
fn(x)
torch.cuda.synchronize()
return (time.perf_counter() - start) / iters
eager_ms = bench(model) * 1000
compiled = torch.compile(model)
compiled(x) # trigger compilation before the timed run
compiled_ms = bench(compiled) * 1000Run both bench() calls back to back on the same instance and the gap between eager_ms and compiled_ms is whatever fusion win TorchInductor actually found in your model's op chain, not NVIDIA's sum(abs(x)) case. A 20-minute session is enough to get a stable read on both numbers, and on a per-minute-billed H100 at $2.65/hr on-demand, that session costs a small slice of a single hour rather than a full reserved block sitting idle while you wait on the compile step to warm up.
Worth saying plainly: the throughput gain from kernel fusion comes entirely from the software stack a tenant runs, PyTorch's Inductor backend, Triton, vLLM, TensorRT-LLM, not from anything a GPU host provides. Spheron rents the hardware; it doesn't compile or fuse a tenant's kernels for them. And a team that has already captured its realistic fusion gains and is genuinely compute-bound gets nothing further from switching rental providers on the strength of fusion alone. At that point the lever is a faster GPU tier, not a cheaper hour.
Run that toggle test on the same GPU class the published numbers were measured on, since a 3x fusion win on one SKU's memory bandwidth profile doesn't automatically transfer to another. Spheron's H100 rental and A100 rental pages cover the exact SKUs most published fusion benchmarks, including the RTX 4090 example above, are run against or scale from.
Every fused kernel still has to run somewhere, and the GPU-hours it saves only count if you're not paying for idle capacity while you measure it.
Frequently Asked Questions
Kernel fusion is combining multiple separate GPU operations into a single kernel launch. Instead of each operation writing its result to HBM and the next operation reading it back, the fused kernel keeps intermediate values in on-chip registers or shared memory and writes only the final result. This cuts both the HBM traffic between ops and the fixed per-launch overhead of the CUDA driver, which runs on the order of 10 microseconds per launch according to Modal's GPU glossary.
Yes. torch.compile's default backend, TorchInductor, applies fusion passes automatically when you wrap a model or function with torch.compile(). It fuses pointwise and reduction operations that sit next to each other in the compute graph (vertical fusion) without requiring you to write a custom kernel. It won't always match a hand-written Triton or CUTLASS kernel on a workload with unusual memory access patterns, but it recovers most of the win for typical elementwise and normalization chains.
No. Fusion pays off on memory-bound operations, where the bottleneck is HBM bandwidth rather than compute, which describes most elementwise, normalization and activation ops in a transformer. It does little for compute-bound kernels like large matrix multiplications, where the GPU's tensor cores are already the constraint and memory traffic isn't. Fusion can also hurt a kernel that relies heavily on atomic operations, since serializing those atomics inside one larger kernel can undo the gain from fewer launches.
It depends entirely on the operation. NVIDIA's own worked example on an RTX 4090 fused a two-kernel sum(abs(x)) reduction and cut execution time from 3.51 milliseconds to 1.18 milliseconds, roughly a 3x improvement, by reducing HBM traffic from 3GB to 1GB. Published results across memory-bound fusion patterns generally land in a 1.5x-3.13x range; atomic-heavy or already compute-bound kernels see far less, sometimes nothing.
