Memory Bound vs Compute Bound: Roofline Model for LLM Inference

"Memory bound" and "compute bound" get thrown around as adjectives, applied to a workload after the fact, usually as an excuse for why adding GPUs didn't help. They're not adjectives. They're two regions of a chart, split by one number, and you can compute that number for your own GPU and your own workload before you rent anything. This post plots it: an NVIDIA H100's own FLOPS and bandwidth, the ridge point where the two meet, and prefill and decode dropped onto the chart as real dots at a real batch size, so "memory bound vs compute bound" stops being a guess and starts being arithmetic.

TL;DR: Memory Bound vs Compute Bound for LLM Inference

• Prefill: processing the prompt as one batched matmul pushes arithmetic intensity into the thousands of FLOPs/byte, so prefill stays compute-bound at nearly any batch size.
• Decode: per the jax-ml/Google DeepMind scaling book, an H100 needs ~280 tokens of batch size in bf16 to cross the ridge, so serving batches of 1-32 stay memory-bound.
• What to shop for: memory-bound workloads need bandwidth per dollar, not FLOPS per dollar. H100 SXM5 on Spheron bills per minute, so you can test the verdict against your workload with no commitment.

What "Memory Bound vs Compute Bound" Actually Means

A GPU does two things when it runs a matrix multiplication: it moves bytes from HBM into the chip, and it does floating-point math on those bytes once they're there. Those two operations have separate hardware ceilings. Peak FLOPS caps how much math the Tensor Cores can do per second. Peak bandwidth caps how many bytes HBM can hand over per second. Every kernel you run is capped by whichever ceiling it hits first.

Compute-bound means the Tensor Cores are the limit: the GPU spends its time doing math, and the memory system has bytes to spare. Memory-bound means HBM is the limit: the Tensor Cores sit partly idle, waiting on the next load, and adding more FLOPS to the chip would do nothing for that kernel. The single number that tells you which one you're in is called arithmetic intensity: FLOPs performed divided by bytes moved to perform them. Compare that number to a GPU's ridge point and the answer stops being a vibe.

The Roofline Chart: One Diagram, Two Axes

The roofline model, first formalized for general-purpose HPC kernels and now the standard way to reason about Tensor Core workloads, plots arithmetic intensity (FLOPs/byte) on the x-axis and achievable throughput (FLOPs/second) on the y-axis, both on log scales. Every kernel is a single point on that chart. Where it lands tells you, at a glance, which hardware ceiling is holding it back.

The Two Roofs: Peak FLOPS and Peak Bandwidth

Two lines bound the chart. A flat line runs across the top at the GPU's peak FLOPS, the compute roof: no kernel can exceed it, no matter how much data it has to work with. A sloped line runs up from the origin at a rate set by peak memory bandwidth, the bandwidth roof: throughput along that line is arithmetic intensity multiplied by bandwidth, since more bytes moved per second at a fixed FLOPs/byte ratio means more FLOPs per second. An NVIDIA H100 SXM5 defines both roofs:

That FP8 figure is worth flagging on its own, because it gets misquoted constantly. NVIDIA's H100 spec sheet lists 1,979 TFLOPS as the number most people copy as "H100 peak," but that's the dense FP8 figure. Mixing those two up is a common way a roofline calculation for an H100 ends up off by 2x before you've even gotten to arithmetic intensity.

The Ridge Point: Where the Roofs Meet

The two roofs intersect at one arithmetic intensity value, the ridge point, and it's just the compute roof divided by the bandwidth roof:

• Dense BF16 ridge point: 989.5 TFLOPS / 3.35 TB/s ≈ 295 FLOPs/byte
• Dense FP8 ridge point: 1,979 TFLOPS / 3.35 TB/s ≈ 591 FLOPs/byte

Below the ridge point, a kernel's achievable throughput is capped by the sloped bandwidth roof: throughput = arithmetic intensity × bandwidth, and no amount of extra compute silicon changes that, because the Tensor Cores are already waiting on data. Above the ridge point, throughput flattens out at the compute roof, and no amount of extra bandwidth changes that, because the chip is already saturated with math. The ridge point is the one number that separates "buy more bandwidth" from "buy more FLOPS," and it's fixed by the GPU, not the model.

Arithmetic Intensity of Prefill vs Decode

LLM inference runs two structurally different phases, and they land on opposite sides of the ridge point almost by construction. Here's the math for each, in the same FLOPs/byte units as the roofs above.

For a linear layer with weight matrix of size d_in × d_out, processing a batch of B tokens in one matmul, in BF16 (2 bytes per weight):

• FLOPs performed ≈ 2 × B × d_in × d_out
• Bytes moved (weight-dominated, loaded once and reused across the batch) ≈ 2 × d_in × d_out

Divide one by the other and the matrix dimensions cancel out entirely: arithmetic intensity ≈ B, the batch size in tokens. That single cancellation is the whole reason prefill and decode end up on opposite sides of an H100's ridge point.

Why Prefill Is Compute-Bound: The Whole Prompt Is One Big Batched Matmul

During prefill, the model processes every token of the input prompt in one shot, as a single batched operation. Per the jax-ml/Google DeepMind scaling book on transformer inference, "during prefill, all matrix multiplications are basically always compute-bound. Therefore, simply maximizing hardware utilization or MFU (Model FLOPs Utilization) is enough to maximize throughput-per-chip (cost) and latency (in the form of TTFT)." A prompt of 2,048 tokens gives an arithmetic intensity around 2,048 FLOPs/byte for the matmul-heavy layers, which is roughly 7x past the H100's dense BF16 ridge point of 295. Longer prompts push it further right on the chart. There's effectively no realistic prompt length short enough to pull prefill's matmuls back below the ridge.

Why Decode Is Memory-Bound: One Token, the Full KV Cache, Every Step

Decode generates one token per step, per sequence, but attention still has to read every previous token's key and value vectors out of the KV cache to compute that one step. FLOPs and bytes moved both scale with the KV cache size, so they grow together and the ratio barely moves. As the same source puts it: "We're doing a tiny amount of FLOPs while loading a massive KV cache. So we're basically always memory bandwidth-bound during attention!" Attention's arithmetic intensity stays close to 1 FLOP/byte almost regardless of batch size, which is the same conclusion FlashAttention's kernel design starts from: standard attention's own memory traffic, not its math, is what a fused kernel is built to cut.

The feedforward and projection matmuls in decode behave differently from attention, since B in the B ≈ arithmetic intensity relationship above is literally the decode batch size (how many sequences you're serving in parallel), not the KV cache length. That's the number worth plotting.

Plotting Real Numbers: H100 SXM5, a 70B Model, Batch 1 vs Batch 32

Put four points on the same chart, using an H100 SXM5's dense BF16 ridge point of 295 FLOPs/byte as the reference and achievable throughput = arithmetic intensity × 3.35 TB/s in the memory-bound region:

The batch 1 and batch 32 dots both sit hard against the sloped bandwidth roof, nowhere near the flat compute roof above them. A 70B-class model serving one request at a time uses less than half a percent of an H100's dense BF16 compute roof, and raising the batch to 32, a serving configuration teams treat as "already batched," still only reaches 11%. The jax-ml/Google DeepMind scaling book's own number for the critical batch size, where decode's matmuls finally cross the ridge point, is roughly 280 tokens on H100 in bf16, which lines up almost exactly with the 295 FLOPs/byte the H100's own spec sheet gives you. Interactive serving essentially never reaches a batch that large per forward pass; that headroom is what continuous batching exists to claim, by packing more concurrent sequences into each decode step instead of leaving the bandwidth roof under-used.

Prefill's row is capped at the compute roof rather than showing its true 2,048 FLOPs/byte position, because that's what the flat part of the roofline means: once you're past the ridge, more arithmetic intensity doesn't buy more throughput. The GPU is already saturated with math.

Memory Bound vs Compute Bound: What to Shop for

Once you know which side of the ridge point your workload lands on, the shopping list writes itself, and it's a different list on each side.

If you're compute-bound (prefill-heavy, large batch, offline scoring jobs): compare GPUs on dense FLOPS, in the precision you'll actually run. This is where FP8 pays off directly, since the dense FP8 compute roof (1,979 TFLOPS on H100) is double the BF16 roof, and a compute-bound kernel's throughput scales with that roof. Batch-tolerant, interruption-tolerant workloads like offline prefill scoring are also a reasonable fit for spot pricing, since a restart just re-runs a batch rather than dropping a live user session.

If you're memory-bound (interactive decode at typical serving batch): compare GPUs on bandwidth per dollar, not FLOPS per dollar, because the compute roof isn't your constraint. Our HBM3e vs HBM4 vs HBM4e guide breaks down the bandwidth-per-dollar math across H100, H200, and B200 in more depth than fits here. Don't guess at your own arithmetic intensity either: Nsight Compute's roofline view plots your actual kernels against the actual roofs, which catches cases the back-of-envelope math above glosses over, like KV cache reads that don't perfectly overlap with weight loads. This post stops at the diagnosis; for the practical fix once you've confirmed you're memory-bound, from KV cache tiering to picking a GPU by MBU%, see our guide to the memory wall problem.

When FP8 or Bigger Batches Change the Answer

FP8 is worth a specific caution here, because the intuitive story ("FP8 doubles your FLOPS, so it makes you more compute-bound") isn't quite right for decode. FP8 weights are stored in 1 byte instead of BF16's 2, so for the same batch size, decode's matmul arithmetic intensity roughly doubles too, since half as many bytes move for the same FLOPs. The compute roof also roughly doubles, from 989.5 to 1,979 TFLOPS. Both sides of the ratio move together, so the critical batch size needed to cross the ridge barely shifts. What FP8 actually buys a memory-bound decode step is faster time-to-token at the same batch, because achievable throughput in the memory-bound region is arithmetic intensity times bandwidth, and arithmetic intensity just doubled. FP8 quantization is a real lever for decode cost, but it's a bandwidth-efficiency lever, not a "now you're compute-bound" lever, and it's worth checking accuracy trade-offs before assuming it's free.

The batch-size lever, by contrast, genuinely can flip the verdict: push decode batch size high enough, generally by serving more concurrent sequences rather than one bigger request, and you cross the ridge point for real. That's the practical reason batching schedulers exist: they're the mechanism that moves a workload's dot from the far-left, deeply memory-bound corner of the chart toward the ridge, without changing anything about the model itself.

The roofline model turns "are we memory-bound" from a guess into a plot you can rebuild for your own model and batch size. Rent GPU compute to test your workload against this plot and adjust next week if your batch size changes.

Get started on Spheron →