LLM Inference Latency Optimization Technique: Compile a Megakernel With Mirage (2026)

Compiling a megakernel is an LLM inference latency optimization technique you can actually reproduce yourself, not just read about. Every LLM inference engine you've deployed, vLLM, TensorRT-LLM, SGLang, issues a separate CUDA kernel launch for nearly every operator in a transformer forward pass: one for the QKV projection, one for RoPE, one for attention, one for the output projection, one for RMSNorm, and so on, repeated at every decode step. Mirage Persistent Kernel (MPK) takes a different approach: it compiles the entire forward pass into one fused GPU kernel, a megakernel, and this post walks through actually building it, from a clean rented GPU to a compiled Qwen3-8B megakernel, then shows you exactly what to measure to see what changes and what doesn't.

TL;DR: How Do You Compile an LLM Into a Megakernel With Mirage?

• The tax: kernel-per-operator dispatch cost ~14.6% of decode time in a 2026 Qwen2.5-1.5B TensorRT-LLM measurement (1,655,550 launches, ~3.3s), via Nsight Systems.
• The fix: Mirage Persistent Kernel (MPK), from CMU's Catalyst group, fuses an entire LLM forward pass into one GPU kernel instead of hundreds of launches.
• The build: git clone --recursive --branch mpk https://www.github.com/mirage-project/mirage && pip install -e . -v.
• The run: python demo/qwen3/demo.py --use-mirage compiles Qwen3-8B into a megakernel; add --profiling to trace it.
• Where it holds: MPK's benchmark suite spans A100, H100 and B200 at batch 1-16; gains are largest in memory-bound, low-batch decode.
• The instance: a Spheron bare-metal GPU, full root access included. Compare GPU pricing →

Why Kernel Launch Overhead and HBM Round-Trips Are the Target of This LLM Inference Latency Optimization Technique

Before touching Mirage, it's worth being precise about what a megakernel is actually fixing, because two distinct costs get lumped together under "kernel overhead" and only one of them is solved by the fix most teams already have in production.

Kernel-Per-Operator Dispatch: The ~14.6% Tax on Decode

A standard inference stack treats each operator in the model graph as its own launch. The CPU issues a kernel, the driver queues it, the GPU executes it, and the CPU issues the next one. Each of those launches carries fixed overhead, driver-side bookkeeping and queue management, that has nothing to do with the actual math the kernel performs. Prefill amortizes this fine, because each kernel does a lot of work per launch. Decode doesn't: an autoregressive step processes one new token per sequence, so the amount of real compute per kernel is tiny relative to the fixed dispatch cost sitting on top of it.

The Ada-MK paper, a separate megakernel project out of Baidu, profiled exactly this on Qwen2.5-1.5B running under TensorRT-LLM, using Nsight Systems to trace every launch. It counted 1,655,550 kernel launches over the run, consuming about 3.3 seconds of total inference time, roughly 14.6% of end-to-end time going to dispatch rather than computation. That's not a rounding error. It's a sixth of your decode latency budget spent on overhead the model architecture doesn't require, purely an artifact of how the engine schedules work onto the GPU.

Inter-Operator HBM Round-Trips: The Gap CUDA Graphs Doesn't Close

If you've already adopted torch.compile and CUDA Graphs for LLM inference, you've already attacked part of this problem. CUDA Graph capture records a sequence of kernel launches once during warm-up and replays the whole graph as a single dispatched unit on every subsequent step, which is exactly the right fix for the CPU-side launch overhead described above.

What it doesn't fix is what happens between those kernels. Each operator in the captured graph is still a separate kernel that writes its output to HBM and the next operator reads it back from HBM before it can start. RMSNorm writes its output, the next matmul reads it. Attention writes its output, the residual add reads it. None of that traffic is model compute; it's the tax of keeping every operator as an independently scheduled unit, and it's a direct consequence of the memory wall problem that already governs decode-phase inference: the GPU spends more time moving bytes between HBM and its compute units than it spends computing on them.

Step-by-Step: Compiling Qwen3-8B Into a Megakernel With Mirage (MPK) on a Rented GPU

MPK ships a runnable demo for Qwen3-8B in its repository, which makes this one of the more directly reproducible megakernel setups available right now. Here's the full path from a bare GPU to a compiled megakernel.

Provisioning a Bare-Metal GPU Instance With Root Access

MPK's CUDA backend is a from-source build: no pre-built wheels were available at the time of writing, according to the project's own setup instructions. That means nvcc has to actually compile CUDA source against your driver and toolkit, which is not something a locked-down managed inference endpoint will let you do. You need a host where you can install packages, write to system paths, and touch the CUDA toolkit directly.

This is the build to run on a Spheron bare-metal GPU instance. Spheron's own pricing page describes its instances as shipping with a dedicated IP address and full root access on both bare-metal and VM options, billed at hourly rate, which matches what a one-off from-source build and benchmark run actually needs: provision, build, measure, tear down, without paying for an idle hourly block in between. The catalog covers both H100 and A100, which happen to be two of the three GPU generations MPK's own benchmark suite targets.

One thing worth checking before you start rather than assuming: neither the pricing page nor the catalog description confirms the CUDA toolkit ships pre-installed on the base image. Run nvcc --version as your first command after SSHing in. If it's missing, install it from NVIDIA's own CUDA toolkit repository for your driver version before you touch the Mirage build; skipping this step is the most common way a "clone and build" guide fails silently twenty minutes in. Spheron's own docs cover instance provisioning and API access if you're scripting this rather than doing it by hand.

Cloning and Building the mpk Branch

MPK's persistent-kernel work lives on a dedicated branch, not main. Clone it recursively, since the build depends on submodules that a shallow, non-recursive clone will silently skip:

git clone --recursive --branch mpk https://www.github.com/mirage-project/mirage
cd mirage
pip install -e . -v
export MIRAGE_HOME=$(pwd)

The -v flag on pip install is worth keeping even though it's noisy. A CUDA extension build failing partway through produces a wall of compiler output, and you want to see where it stopped rather than get a generic "build failed" at the end. MIRAGE_HOME needs to be set in whatever shell or session actually runs the demo afterward, so if you're running this inside tmux or screen, set it in that session, not just the one you cloned in.

Running the Qwen3 Demo: Eager vs --use-mirage

With the build done, the repository's demo/qwen3 directory gives you a same-model, same-hardware comparison without writing any harness code yourself. Run the eager PyTorch baseline first:

python demo/qwen3/demo.py

Then run the same model compiled into a megakernel:

python demo/qwen3/demo.py --use-mirage

The first --use-mirage invocation pays a compile step before it starts generating, the same general shape as a torch.compile cold start: MPK has to lower the model graph into its SM-level representation and generate the fused kernel before it can run it. That cost is one-time per process, not per request, so it's a fixed setup tax you pay once per deployment, not something that shows up in your steady-state decode latency.

Profiling the Megakernel With --profiling

To actually see what the fused kernel is doing rather than just timing the wall clock around it, add the profiling flag:

python demo/qwen3/demo.py --use-mirage --profiling

Read the resulting trace the same way you'd read any Nsight-captured CUDA trace: look at how much of the kernel's wall time is spent in compute versus waiting on memory, and compare that occupancy picture against the eager run's per-operator traces if you captured those separately. This is also the step where a claim like "the megakernel removed the HBM round trips" stops being an architectural description and becomes something you can actually verify against your own hardware.

What to Measure: Per-Token Latency Before and After, and Where the Gain Breaks Down

The metric that matters here is per-token decode latency, the same inter-token latency figure that shows up in any TTFT and ITL latency budget: wall-clock time for the decode phase divided by tokens generated. Both the eager and --use-mirage demo runs give you that number directly, on the same GPU, same model weights, same prompt, so the comparison you get out of this setup isn't cross-hardware or cross-paper, it's your own run against itself, timed on the instance you provisioned.

Decode Latency at Batch Size 1 vs Larger Batches

At batch size 1, decode is about as memory-bound and underutilized as GPU work gets: each step is one new token per sequence, the matmuls are thin, and the fixed cost of a kernel launch is large relative to the actual math happening inside it. This is exactly the regime the launch-overhead and HBM-round-trip problems described above hit hardest, and it's where fusing everything into one megakernel has the most overhead to remove. Run the eager and --use-mirage commands above at batch 1 and the gap between them is the clearest signal you'll get from this whole exercise.

Push batch size up, toward the 1-to-16 range MPK's own benchmark suite tests across A100, H100 and B200, and the picture shifts. Each decode step now does proportionally more real compute per kernel launch, because you're computing that step for more sequences at once, while the fixed per-launch overhead doesn't grow with batch size. The dispatch tax that was a large fraction of a small number becomes a smaller fraction of a larger number. The megakernel still removes the same launches and the same HBM round trips, but there's more genuine compute sitting underneath them to compare against, so the relative gain compresses. This is the honest caveat the paper's own batch-size sweep exists to surface, not something a megakernel-specific limitation invents.

Where the Speedup Tapers: Compute-Bound and Multi-GPU Regimes

Two regimes narrow the gain further, for reasons that follow directly from what a megakernel actually fixes:

None of these mean a megakernel does nothing outside single-GPU, low-batch decode. It means the specific problem MPK is built to solve, launch dispatch and inter-operator HBM traffic, is a smaller share of total latency once one of those other costs starts to dominate. Measure your own workload's batch size and sequence length before assuming the gain you'd see at batch 1 carries over to your production traffic shape.

Megakernel vs CUDA Graphs: Overlapping Fixes, Different Ceilings

CUDA Graphs and a Mirage megakernel are not competing solutions to the same problem; they target different halves of the 14.6% dispatch tax and the HBM round-trip cost sitting alongside it. CUDA Graphs replays a captured launch sequence as one dispatched unit, which removes CPU-side scheduling overhead cheaply and is already production-stable in PyTorch 2.6. It does not touch what happens between the kernels in that sequence: each operator still writes to and reads from HBM independently. A megakernel fuses the operators themselves, so there's no intermediate HBM round trip to eliminate in the first place, which is a higher ceiling on paper.

That higher ceiling comes at a real engineering cost, which is precisely why MPK exists as a compiler rather than a coding pattern. CUDA Graphs is something you can adopt in an afternoon by wrapping your existing forward pass. A megakernel needs a compiler to generate correct, fused GPU code from your model graph, which is the entire reason a project with more than 20 co-authors across five institutions exists to build one, instead of every serving team hand-rolling it themselves.

When a Megakernel Is (and Isn't) the Right LLM Inference Latency Optimization Technique

Compile a megakernel when your latency budget is dominated by single-GPU, low-batch decode on a model and shape you're not swapping out frequently, since the compile step is a fixed cost you want to pay once and amortize, not something you want to re-pay every time your traffic shape changes. It's the right fix when you've already exhausted the cheaper options, quantization, continuous batching, a CUDA-Graphs-based serving stack, and you're still latency-bound on dispatch and HBM traffic rather than raw compute.

It's the wrong fix, or at least not the first one, when your workload runs at higher batch sizes where compute already dominates dispatch overhead, when you're serving many different model shapes and can't amortize a compile step across enough requests, or when your latency is actually dominated by multi-GPU collective communication rather than single-GPU kernel scheduling. In any of those cases, start with the broader vLLM vs TensorRT-LLM vs SGLang decision framework to confirm where your latency is actually going, rather than jumping straight to a compiler-level fix this specific. A megakernel is a precise tool for a precise, well-diagnosed problem, not a default you reach for before you've profiled.

Building MPK from source needs root access most managed inference platforms won't give you. Spheron's bare-metal and VM GPU instances ship with full root access and per-minute billing, so you can provision, build, and measure without committing to an hourly block.

Spheron H100 →