Run a normal transformer layer on an NVIDIA H100 SXM and you will not see the peak BF16 Tensor Core throughput printed on its spec sheet. Not because the GPU is broken, and not because your code is bad. The number was never describing your workload in the first place.
This is the recurring gap in every GPU spec sheet vs real world performance comparison: the printed peak is a best-case ceiling built from a specific combination of sparsity, matrix shape, and workload phase that almost nothing in production actually matches. Once you know what physically has to happen for a tensor core instruction to fire, and what the roofline model says about why prefill and decode behave like two different chips, the spec sheet stops being a marketing number and starts being a tool you can actually read.
GPU Spec Sheet vs Real World Performance: Peak TFLOPS Is a Marketing Number
Every GPU vendor publishes one headline TFLOPS figure per precision, and every one of those figures is the best case, not the expected case. It assumes a specific sparsity pattern, a workload shaped to avoid every inefficiency the hardware has, and zero time spent waiting on memory. Real serving traffic gets none of those for free.
The Sparsity Trick: Why the Headline Number Is 2x the Real One
NVIDIA's developer documentation on Ampere and Hopper structured sparsity spells out the pattern: "Among each group of four contiguous values, at least two must be zero, which is a 50% sparsity rate." That's the whole trick. As the same documentation puts it, "the sparse Tensor Cores process only the non-zero values when doing matrix multiplication and theoretically, the compute throughput would be 2x compared to the equivalent dense matrix multiplication." That "theoretically" is doing real work: it's the reason the number on the box and the number in your logs are different figures.
Getting there isn't free, either. Skip it, which most inference deployments do, and you're running dense weights on a Sparse Tensor Core that has nothing to skip. You get the dense rate, not the sparse one, meaning you're capped well under half of the headline figure the spec sheet advertises.
What "Peak" Silently Assumes (and Never Ships With)
Sparsity is the biggest single discount, but it's not the only one baked into the number. The peak figure also assumes:
- Every matrix dimension is shaped for the hardware. cuBLAS and cuDNN fall back to slower tiling paths on odd shapes, covered in the next section.
- The workload is purely compute-bound. No stalls waiting on HBM, no kernel launch overhead, no data movement between operations.
- One precision, one operation, running continuously. Real inference mixes attention, normalization, and routing kernels of very different shapes and costs, not one uninterrupted GEMM.
- A single instant, not a sustained interval. DCGM samples utilization as an average over a window; a burst that hits 100% for ten milliseconds and idles for the next ninety is a very different chip from one holding 90% continuously, even though both can appear in the same "peak" conversation.
None of these assumptions are hidden. They're just never printed next to the number.
What Actually Triggers Tensor Cores
A tensor core doesn't fire just because a GPU kernel launches. It fires on a specific matrix-multiply-accumulate instruction, issued against operands that satisfy the hardware's precision, shape, and memory layout requirements. Miss any of those and the same matmul either falls back to slower CUDA cores or runs through tensor cores at a fraction of their rated throughput.
Precision, Matrix Shape, and Alignment: The Fine Print That Decides If You Even Get Tensor Core Paths
NVIDIA's own performance guide for fully connected layers states the requirement plainly: "Choose the batch size and the number of inputs and outputs to be divisible by 4 (TF32) / 8 (FP16) / 16 (INT8) to run efficiently on Tensor Cores." cuBLAS and cuDNN will still execute a GEMM with dimensions that don't hit those multiples, but they route it through a less efficient tiling strategy instead of the fast path.
This matters more than it sounds like it should, because transformer shapes are rarely tidy round numbers by accident. Vocabulary sizes, head counts, and hidden dimensions come from architecture decisions made for reasons that have nothing to do with tensor core alignment, and quantization or padding choices downstream can shift a shape just enough to fall off the fast path without anyone noticing in a profiler that only reports GPU-busy time.
Compute-Bound vs Memory-Bound: Reading the Roofline
The roofline model puts a hard ceiling on what any workload can achieve on a given chip, and the ceiling depends on arithmetic intensity, the ratio of floating-point operations to bytes moved from memory. Below the ridge point, a workload is memory-bound and adding more FLOPS does nothing, because the chip is waiting on data, not compute. Above it, the workload is compute-bound and the FLOPS ceiling is what actually limits you.
The ridge point itself is chip-specific. ZeroEntropy's engineering explainer on arithmetic intensity puts it plainly: "An H100's ridge sits near 295 FLOPs/byte at FP16; an A100 around 200; consumer cards somewhat lower." A workload with arithmetic intensity below that line will never reach peak TFLOPS on that chip, full stop, no matter how well the kernel is written.
This is exactly why the same GPU can post wildly different utilization numbers on two halves of the same inference request, which is the subject of the next section.
Measuring Real Utilization on a Serving Workload
The short answer: GPU-Util from nvidia-smi tells you whether any kernel was running on the streaming multiprocessor. It does not tell you whether that kernel was using the tensor cores, or how much of the tensor pipe's capacity it consumed. For that you need DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, and you need to look at it separately for prefill and decode, because on an LLM serving workload those two phases are not the same measurement.
The Metric: DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, Not GPU-Util
NVIDIA's own DCGM documentation defines the field as "the fraction of cycles the tensor (HMMA / IMMA) pipe was active," and adds that "the value represents an average over a time interval and is not an instantaneous value." The scale reading matters too: "an activity of 1 (100%) is equivalent to issuing a tensor instruction every other cycle for the entire time interval."
That averaging is a real limitation, not a footnote. A mid-range reading, say 50%, can mean two very different things on the ground: every streaming multiprocessor busy exactly half the time, or half the SMs pegged at 100% while the rest sit idle. DCGM alone can't tell you which. You need it alongside occupancy and SM-active counters, and ideally a kernel-level trace from Nsight Compute, before you treat a tensor-active number as a diagnosis rather than a symptom.
Pulling the number in the first place takes one command, straight from DCGM's own tooling reference: dcgmi dmon -e 1001,1004,1005. The columns come back as GRACT, TENSO, DRAMA. Point that same command at a compute-bound prefill batch and you should see the same shape. Point it at a decode-heavy window instead and TENSO is the column that drops while GRACT barely moves, because the SM is still scheduling work, just not tensor-core work. That's the difference between GPU-Util and tensor utilization, visible in one dmon output instead of argued about in the abstract. The GPU goodput engineering guide walks through pairing DCGM counters with framework-level metrics like vLLM's cache and request counters to tell those cases apart, and the GPU monitoring guide covers wiring DCGM Exporter into Prometheus and Grafana if you don't already have the pipeline running.
Prefill vs Decode: Two Utilization Numbers on the Same Chip
Prefill and decode are the same GPU, running the same model, on two structurally different problems. Prefill processes the whole prompt as one large batched matmul: high arithmetic intensity, compute-bound, and squarely above most GPUs' roofline ridge. Decode generates a single token per step against the full KV-cache: low arithmetic intensity, memory-bound, and stuck below the ridge no matter how the kernel is tuned.
The measured spread on real hardware is large. A teardown of prefill and decode on H100 reports that on an H100 SXM, "prefill achieves 200-400 arithmetic operations per byte of memory accessed," with "utilization sitting between 90% and 95%." The same analysis gives a concrete example: "A Llama 70B model running inference on an H100 GPU hits 92% compute utilization during prefill. Thirty milliseconds later, during decode, that same GPU drops to 30%." Same GPU, same request, roughly a 3x drop in tensor core activity thirty milliseconds apart.
That's the failure mode vLLM's chunked prefill feature exists to blunt. As the project's own optimization docs put it, chunked prefill "allows vLLM to process large prefills in smaller chunks and batch them together with decode requests," specifically to help achieve "better GPU utilization by locating compute-bound (prefill) and memory-bound (decode) requests to the same batch." It doesn't turn decode into a compute-bound phase, nothing does, but it stops a long prefill from monopolizing the GPU while every in-flight decode request stalls behind it. The vLLM production deployment guide covers the tensor-parallel and FP8 configuration this runs on top of.
GPU Spec Sheet vs Real World Performance: The Numbers Stacked Up
Stack the numbers above next to each other and the discount from "headline peak" to "what you actually get" compounds fast. Each row is an answer to the same underlying question: what fraction of the number on the spec sheet does a real workload convert into useful work?
| Stage | Figure | What it represents |
|---|---|---|
| Printed peak (H100 SXM, BF16) | 1,979 TFLOPS | Assumes 2:4 sparsity, per NVIDIA's own datasheet footnote |
| Prefill, dense weights, aligned shapes | 90-95% tensor-pipe utilization | Compute-bound phase, measured on H100 SXM with a Llama 70B, 4,096-token prompt |
| Decode, same chip, same request | 20-40% tensor-pipe utilization | Memory-bound phase, same H100 SXM |
Alexander Rocha, founding engineer at ZeroEntropy, frames the size of that gap directly in the site's MFU explainer: "A 5x MFU gap between naive and well-engineered code is normal. That gap is most of the operating cost of a training run." Read that against the roofline math above and the conclusion holds for inference too: the difference between a naive deployment and a tuned one isn't a rounding error on your GPU bill, it's most of the bill.
Rented hardware turns that percentage gap into a dollar figure, and this is where the math stops being abstract. Our own live pricing page lists on-demand H100 at $2.65/hr as of this writing, quoted the same way every vendor quotes it: dollars per GPU-hour, with no mention of what fraction of that hour the tensor pipe actually spent doing tensor-core work. Divide that rate by the utilization each phase delivers and the same GPU-hour buys very different amounts of tensor-core throughput. At 92% utilization during prefill, you're paying roughly $2.88 for an hour of fully-utilized tensor-core work. At 30% during decode, on the same chip, the same math puts the effective rate near $8.83. Nobody bills decode separately from prefill in practice, so treat this as a way to see the gap rather than a real line item, but it's the calculation a buyer should run before assuming a cheap advertised $/hr always means cheap tokens.
Pricing fluctuates based on GPU availability. The price above is based on 03 Sep 2026 and may have changed. Check current GPU pricing → for live rates.
If you want a third-party check on any of this rather than taking a vendor's word for it, MLPerf's audited inference results are the closest thing the industry has to an apples-to-apples measurement across chips and stacks; the MLPerf Inference v6.0 breakdown covers what the latest round showed for H200, B200, and MI355X against their own printed specs.
How to Read a GPU Datasheet Like a Buyer
The practical version of everything above is a short list of questions to ask before a peak TFLOPS number changes a buying decision:
- Is the number sparse or dense? If a datasheet lists one Tensor Core figure per precision with no separate dense row, assume it's the sparse number and divide accordingly for anything you're not deliberately pruning into a 2:4 pattern.
- What workload was it measured on? A GEMM microbenchmark with hand-picked shapes tells you almost nothing about a transformer layer with your actual hidden dimension and batch size.
- Compute-bound or memory-bound? Ask where the workload sits relative to the chip's roofline ridge point. If it's decode, or anything else with low arithmetic intensity, no amount of peak TFLOPS will move the needle; bandwidth is the ceiling.
- Ask for DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, not GPU-Util. A vendor quoting "95% GPU utilization" has told you the SM was busy. It hasn't told you the tensor cores were the reason. Ask for the tensor-pipe-active number specifically, sampled across both prefill and decode if the workload is LLM serving.
- Ask for MFU on a real training run, not a microbenchmark. A vendor citing a near-peak MFU number with no methodology behind it deserves the same scrutiny you'd give any other outlier claim: the 5x gap Rocha describes between naive and well-engineered code above doesn't close on its own, so ask what specifically got tuned to reach that number.
None of this means the spec sheet is useless, it's still the right starting point for comparing architectures. It means treating the printed TFLOPS figure as a ceiling to discount from, not a number to expect, and asking for the measurement that actually describes your workload before you size a fleet around someone else's best case. For a broader comparison across current-generation chips on the metrics that matter for buying, the GPU cloud benchmarks page tracks throughput, specs, and pricing side by side, and the best GPU for AI inference guide turns the workload-bound-vs-compute-bound question above into a concrete GPU-by-workload recommendation.
The gap between a spec sheet and a served token is exactly what DCGM_FI_PROF_PIPE_TENSOR_ACTIVE is built to expose. Size a deployment against a measured number, not a headline TFLOPS figure.
Frequently Asked Questions
The headline TFLOPS number on a spec sheet assumes 2:4 structured sparsity (roughly double the dense figure), perfectly aligned matrix dimensions, and a compute-bound workload with no memory stalls. Most inference traffic fails at least one of those conditions: weights are usually dense, shapes are not always clean multiples of 8 or 16, and autoregressive decode is memory-bound by construction. Each unmet assumption costs real throughput, which is why measured utilization routinely lands at a fraction of the printed peak.
It is the fraction of GPU cycles during which the tensor (HMMA/IMMA) pipe was active, averaged over the sampling interval, per NVIDIA's DCGM documentation. A reading of 100% is equivalent to issuing a tensor core instruction every other cycle for the whole window. It is a much more direct signal of matmul throughput than DCGM_FI_DEV_GPU_UTIL, which only reports whether any kernel was running on the SM, tensor core or not.
There's no single universal target, but the gap between naive and tuned code is large. Alexander Rocha, founding engineer at ZeroEntropy, puts it directly: a 5x MFU gap between naive and well-engineered code is normal, and that gap accounts for most of the operating cost of a training run. Treat that as a reason to measure your own MFU rather than assume a vendor's peak TFLOPS number translates directly into training throughput.
Prefill processes an entire prompt as one large batched matmul, which is compute-bound and can push tensor core utilization above 90%. Decode generates one token at a time against the full KV-cache, which is memory-bound and typically shows tensor core utilization in the 20-40% range on the same chip, sometimes on the same request a few milliseconds apart.
