Reproducible GPU Benchmark GEMM: CUTLASS Profiler Tutorial

Advertised TFLOPS are a ceiling, not a promise. A reproducible GPU benchmark GEMM setup is how you find out where your own rented H100 or B200 actually sits under that ceiling, instead of trusting a datasheet number you never tested against your own card, driver version, and clock state. This post walks through that setup using NVIDIA's own CUTLASS profiler: how to build it, how to lock clocks so the number holds still, a sweep script you can clone and run today, and how to read GFLOPs/s against the peak your GPU's datasheet claims. If you haven't yet written a custom kernel and want the authoring side of this story, our CUTLASS vs Triton kernel authoring guide covers that; this post is about verifying the throughput of kernels that already exist, whether they came from CUTLASS, cuBLAS, or a serving framework's own kernel library.

TL;DR: A Reproducible GPU Benchmark GEMM Checklist for the CUTLASS Profiler

• Build it once: cutlass_profiler builds from CUTLASS's tools/profiler/ directory against CUDA 12.0+ and CMake 3.18+, targeting -DCUTLASS_NVCC_ARCHS=90a (H100) or 100a (B200).
• Lock clocks first: NVIDIA's TensorRT docs warn floating or throttling clocks cause unstable measurements; run sudo nvidia-smi -lgc <freq> before any sweep.
• Peak is per-shape: Colfax Research measured 630 TFLOP/s against a 750 TFLOP/s peak on an H100 PCIe, about 84% of that config's ceiling, not the card's 989 TFLOPS datasheet peak.
• Initialization moves the number: switching random floats to random +/-1 inputs moved one Colfax kernel from about 530 to 630 TFLOP/s, no code change.
• Run the same sweep on a Spheron bare-metal H100 SXM5 or B200 SXM6 instance. Check current GPU pricing →

Why Vendor TFLOPS Numbers Rarely Match What You Get

A datasheet TFLOPS figure is a theoretical maximum: every tensor core issuing every cycle, no memory stall, no thermal throttling, no launch overhead. That number describes what the silicon can theoretically do, not what any single GEMM call will do on it.

Real kernels give that number up in a few specific, measurable ways.

Clocks are one of the biggest reasons the gap widens further than it needs to. As NVIDIA's Robert_Crovella has explained on the developer forums, "running gemm or tensorcore codes will often cause the GPU to throttle its clocks to stay within an appropriate power envelope". A GEMM-heavy sweep is precisely the workload that pushes a card into that throttling regime, which means the number you get depends as much on your power and clock configuration as on the kernel itself. That is the gap a reproducible methodology has to close before the GFLOPs/s figure means anything.

A Reproducible GPU Benchmark GEMM Methodology, Set Up on a Rented GPU

Getting a number you can trust twice needs three things locked down before you record a single result: the binary, the clock state, and the input data. Get any one of those wrong and two runs of the same kernel on the same card will disagree, sometimes by 15-20%, for reasons that have nothing to do with the code being tested.

Building the CUTLASS Profiler (CUDA Toolkit and CMake Prerequisites)

The CUTLASS profiler is a command-line tool built from the tools/profiler/ directory of the CUTLASS repository, not a prebuilt binary you download. Per NVIDIA's CUTLASS quickstart guide, building it from source requires CUDA Toolkit 11.4 or later (12.0 recommended), CMake 3.18 or later, and a C++17 host compiler, minimum g++ 7.5.0. Target architecture is set via -DCUTLASS_NVCC_ARCHS: use 90a for Hopper (H100), or 100a for Blackwell (B200).

git clone https://github.com/NVIDIA/cutlass.git
cd cutlass
mkdir build && cd build

# H100 (Hopper)
cmake .. -DCUTLASS_NVCC_ARCHS=90a -DCMAKE_BUILD_TYPE=Release

# B200 (Blackwell): swap the line above for
# cmake .. -DCUTLASS_NVCC_ARCHS=100a -DCMAKE_BUILD_TYPE=Release

make cutlass_profiler -j$(nproc)

Building against the wrong arch flag is the single most common reason a fresh CUTLASS checkout profiles a kernel that never uses the tensor core paths you think it does. Build once per architecture, and keep the binary and the CUTLASS commit hash it came from alongside every report you generate.

Locking Clocks Before You Trust a Single Number

GPU clocks that float or throttle mid-run are the single largest source of non-reproducibility in a GEMM sweep, and NVIDIA says so directly in its own benchmarking guidance. The TensorRT documentation states that running workloads with floating clocks or with throttling taking place can lead to more non-determinism in tactic selections and unstable performance measurements across inferences. GEMM sweeps are exactly the sustained tensor-core workload that triggers this, since a long sweep keeps the card hot for minutes at a time rather than seconds.

The fix is a fixed clock, applied before you run anything:

# Lock GPU and memory clocks to a known frequency before profiling
sudo nvidia-smi -lgc <target-clock-mhz>

# ... run your sweep here ...

# Release the lock when you're done
sudo nvidia-smi -rgc

Expect the locked number to read a little lower than whatever a floating or max-boost clock would report. That is the honest tradeoff: a lower, stable number you can reproduce next week beats a higher one you can never get again.

A Reproducible GEMM Sweep Script (Clone and Run)

The CUTLASS profiler's basic invocation is a single GEMM shape: cutlass_profiler --operation=Gemm --m=1024 --n=1024 --k=128. Its real value for benchmarking, though, is the sweep syntax the profiler documentation describes, which lets one invocation cover a whole range of problem sizes: --m=1024:4096:256 --k=128:8192:128 sweeps M from 1024 to 4096 in steps of 256, and K from 128 to 8192 in steps of 128, running every combination against N.

Here is a script that wraps that sweep with the reproducibility controls above baked in, ready to clone onto a rented GPU:

#!/usr/bin/env bash
set -euo pipefail

# reproducible-gemm-sweep.sh
# Usage: ./reproducible-gemm-sweep.sh <clock-mhz> <output-prefix>

CLOCK_MHZ="${1:?Pass a target clock in MHz, e.g. 1500}"
OUT_PREFIX="${2:-gemm-sweep}"
PROFILER=./cutlass/build/tools/profiler/cutlass_profiler

echo "Locking GPU clocks to ${CLOCK_MHZ} MHz..."
sudo nvidia-smi -lgc "${CLOCK_MHZ}"
trap 'echo "Releasing clock lock..."; sudo nvidia-smi -rgc' EXIT

"${PROFILER}" \
  --operation=Gemm \
  --m=1024:8192:512 \
  --n=1024:8192:512 \
  --k=128:8192:128 \
  --A=f16:row --B=f16:column --C=f16:column \
  --warmup-iterations=10 \
  --profiling-iterations=100 \
  --sort-results-flops-per-sec \
  --output="${OUT_PREFIX}.csv"

echo "Done. Top results by GFLOPs/s are at the top of ${OUT_PREFIX}.csv"

Three details make this reproducible rather than merely runnable. It locks and releases clocks around the sweep automatically, including on failure, via the trap. It fixes --warmup-iterations and --profiling-iterations explicitly rather than relying on defaults that could change between CUTLASS releases. And, per the flags NVIDIA's CUTLASS profiler documentation exposes, it writes every result to a CSV via --output, sorted by throughput with --sort-results-flops-per-sec, so the report is a file you can diff against a rerun rather than a number you half-remember from a terminal scrollback.

Reading the Output: GFLOPs/s vs Advertised Peak

The CSV the profiler writes carries the kernel configuration, disposition, runtime, and a GFLOPs/s column for every shape in the sweep. That last column is the number to compare against a datasheet, and the comparison is a single division: measured GFLOPs/s divided by the precision-appropriate peak, expressed in the same units, times 100.

Take Colfax Research's own published result as the worked calculation. Its best CUTLASS-profiler kernel measured 630 TFLOP/s (630,000 GFLOPs/s) against a 750 TFLOP/s dense FP16 peak for that specific M/N/K shape and GPU:

630 / 750 = 0.84 → 84% of peak for that configuration

Two things matter about that division that are easy to miss the first time. First, the peak on the right side of it has to be the peak for the precision and shape you actually ran, not the headline number on the box; a GPU's FP16 peak, FP8 peak, and FP4 peak (where supported) are different numbers, and mixing them up either flatters or damns a kernel that did nothing wrong. Second, a percentage from one shape doesn't transfer to another. CUTLASS's own tuning heuristics pick different kernel configurations for different M/N/K combinations, so a sweep across shapes will show percent-of-peak rising and falling across the CSV, not sitting at one flat number. That variation across the sweep, not any single row, is the thing worth writing down.

Colfax's own result illustrates why the input data matters just as much as the shape. The same multistage kernel measured about 530 TFLOP/s with random floating-point inputs and about 630 TFLOP/s with random +/-1 inputs, a nearly 19% swing from initialization alone, with no change to the kernel under test. Note which initialization strategy you used alongside every CSV you keep, or a rerun months later will look like a regression that never happened.

Running This on Spheron H100 and B200 Instances

The methodology above is only useful once you run it against real hardware rather than reading about someone else's card. Spheron rents bare-metal H100 SXM5 and B200 SXM6 instances in the same catalog, so the identical build-lock-sweep script above runs unmodified across both generations, just swap the CUTLASS arch flag and the target-clock argument. Spheron's own docs describe its instances as delivering bare metal performance rather than a virtualized slice of one, which is what root-level access to nvidia-smi -lgc and to the profiler's hardware counters actually needs; our Nsight Compute profiling guide covers the same permission requirement in more depth for kernel-level profiling, where most shared and serverless GPU platforms fail with ERR_NVGPUCTRPERM for exactly this reason.

H100 SXM5: What to Expect Against the 989 TFLOPS Datasheet Number

The H100 SXM5 datasheet lists 989 teraFLOPS as the dense BF16/FP16 Tensor Core peak, the ceiling every GEMM sweep on an H100 is measured against. Colfax Research's own CUTLASS-profiler run, on an H100 PCIe rather than the higher-clocked SXM5, landed at 630 TFLOP/s against a 750 TFLOP/s peak for that configuration, roughly 84%. That is the calibration point worth running the sweep script against on your own Spheron H100 SXM5 instance: a well-tuned kernel in the low-to-mid 80% range on a comparable shape is a healthy result; a number well below that on a locked clock is worth investigating before you trust anything downstream of it, whether that is a cost-per-token calculation or a serving-framework benchmark. Our GPU cloud benchmarks roundup has the full cross-GPU spec table (memory bandwidth, VRAM, pricing) if you're deciding which card to rent before you get to the profiling stage at all.

B200 SXM6: Running the Same Sweep on Blackwell

B200 is a harder card to calibrate against than H100, and not because the hardware is worse. NVIDIA's current public B200 specification pages report Tensor Core throughput at the level of the full 8-GPU HGX system rather than as a single clean per-GPU dense figure the way the H100 datasheet does, which means turning a system-wide number into a fair per-GPU comparison takes an extra step most readers never do correctly. That is a smaller version of the exact problem this whole post exists to solve: a headline number needs work before it tells you anything about the card in front of you. Rebuild the profiler with -DCUTLASS_NVCC_ARCHS=100a and point the same sweep script at a B200 SXM6 instance on Spheron, and you get a number measured against your own card and your own shapes rather than a system-level figure divided by eight and hoped to be close. If you're weighing B200 against H200 or GB200 for a specific workload rather than benchmarking a card you've already rented, our H200 vs B200 vs GB200 comparison covers that decision directly.

Why Two Runs of the Same Sweep Can Disagree (and How to Stop It)

Three variables account for almost every case of the same kernel binary producing two different GFLOPs/s numbers on two separate runs, and none of them are bugs in the kernel.

Clocks that were never locked. A floating clock lets the card boost during a short warmup and throttle once sustained tensor-core load raises the die temperature mid-sweep, which is exactly the failure mode Robert_Crovella describes and exactly what nvidia-smi -lgc exists to prevent. Run the lock command before every sweep, not just the first one; a reboot or a driver update resets it.

Initialization strategy left unrecorded. Colfax Research's 530-to-630 TFLOP/s swing came entirely from switching random floats to random +/-1 values in the input matrices. Note the initialization flag in whatever report you save, the same way you'd note a git commit hash, because it changes the number as much as the kernel does.

Too few, or too many, profiling iterations. Too few and the timing captures warmup noise; Colfax Research also found that excessive profiling iterations introduced thermal throttling of their own that skewed early results in the run. Fix --profiling-iterations or --profiling-duration explicitly (both are options the profiler exposes directly) rather than leaving it to a default that may change between CUTLASS releases, and keep that number the same across every comparison you make.

Lock all three, and a GEMM sweep from a rented H100 or B200 becomes something you can hand to a teammate, rerun in six months, and trust to mean the same thing both times. That's the actual bar a repeatable GEMM benchmark has to clear, and it has nothing to do with which provider's card you happened to rent. If your measured throughput comes back well under what a comparable kernel should hit even with clocks locked, the next place to look is occupancy: our CUDA occupancy calculator guide walks through why a kernel sitting at 100% occupancy can still run slower than one at 30%.

Advertised TFLOPS numbers are the same on every provider's spec sheet; what you actually get is a function of the exact card, clocks, and driver stack under your workload, which is why it's worth measuring rather than assuming. Spheron rents bare-metal H100 SXM5 and B200 SXM6 instances with no long-term commitment, so you can run this exact sweep before you commit to a longer rental.

Get started on Spheron →