AMD's MI355X posted MLPerf Inference v6.0 numbers within single digits of NVIDIA's B300 in April 2026, and most of the coverage since has repeated that percentage as if it settled the AMD-versus-NVIDIA question. It's an incomplete answer, because MLPerf scores a full inference stack, not the attention kernel that actually decides how many of those tokens a rented GPU turns into revenue per dollar. This is the AMD vs NVIDIA kernel level benchmark 2026 that neither AMD's MLPerf submission nor HipKittens' own paper actually ran: MI355X's attention kernel held against FlashAttention-3 on NVIDIA H200 specifically, using each side's own published, reproducible numbers, with Spheron's live H200 rental standing in for the NVIDIA half.
TL;DR: The AMD vs NVIDIA Kernel Level Benchmark 2026 for MI355X vs H200
- Kernel-level gap: no public test isolates MI355X's attention kernel against H200. MLPerf v6.0's per-GPU throughput puts MI355X within 20% of H200, but that's a system-level number, not the attention op alone.
- Forward attention: HipKittens is "competitive with FlashAttention-3" against NVIDIA B200 (8.0 TB/s bandwidth), a harder target than H200's 4.8 TB/s.
- Backward attention: HipKittens beats AMD's own hand-tuned assembly by 1.2-2.4x on GQA backwards, its clearest kernel-level win so far.
- Pricing, 07 Sep 2026: Spheron rents H200 SXM5 at $4.79/hr on-demand, $2.57/hr spot; the only verified on-demand MI355X rate is Oracle OCI at $8.60/GPU-hr.
- Verdict: AMD's kernel story improved against B200/B300, but the MI355X-vs-H200 attention benchmark still doesn't exist publicly. Compare live H200 rates →
Hypothesis: Does the AMD vs NVIDIA Kernel Level Benchmark 2026 Match the MLPerf Gap?
If MI355X lands within single digits of B300 at the system level, the obvious next question is whether that gap holds, widens, or closes once you strip away batching, KV cache management, and networking and look at one attention op running in isolation. MLPerf is the number every buyer sees first. It is also the number that tells you the least about which kernel is actually doing the work.
What MLPerf v6.0 Actually Measured (System-Level, Not a Single Op)
MLPerf Inference v6.0 results, published by MLCommons on April 1, 2026, put AMD's single-node MI355X submission in the closed division ahead in some scenarios and behind in others: matching NVIDIA's B200 on LLaMA 2 70B in the Offline scenario, running near parity in Server, and exceeding B200 in Interactive. Against B300, the newer and faster of NVIDIA's two current Blackwell parts, single-node MI355X landed at 92% of B300 throughput Offline, 93% in Server, and 104% in Interactive, according to the coverage of AMD's own MLPerf submission.
None of those numbers isolate the attention kernel, and that's true of the H200 comparison too. Our own MLPerf v6.0 explainer normalized the closed division's per-GPU throughput and found MI355X within 20% of H200 on LLaMA 2 70B and GPT-OSS 120B, a real, directly-comparable system-level number that AMD's own press materials don't lead with (those instead frame the submission against B200 and B300). But normalized per-GPU throughput, whichever NVIDIA chip it's measured against, still bundles request scheduling, KV cache paging, quantization, and interconnect behavior into one figure. A GPU can win an MLPerf scenario on scheduling efficiency while running a slower attention kernel underneath, or lose on interconnect overhead while running a faster one. The HipKittens AITER integration we covered in March is part of why that multinode number moved as much as it did, since GEMM and GQA/MHA attention are exactly the operators the framework targets. But "part of why a system score moved" and "how the kernel itself performs" are two different claims, and only one of them is checkable.
The Comparison Nobody's Published: MI355X vs H200, Same Attention Op
Here's the gap this post is trying to close, and it's narrower than "no H200 comparison exists at all." A system-level MI355X-vs-H200 number already exists, from the MLPerf data above. What doesn't exist is an isolated one. HipKittens' own paper states its forward attention kernel is "competitive with FlashAttention-3 under comparable settings" despite the substantial scheduling and hardware differences between AMD's CDNA4 and NVIDIA's Blackwell, but that comparison point is B200. The HipKittens paper never runs the same kernel against H200, and AMD's own MLPerf framing sticks to B200 and B300 too, presumably because those are the parts AMD is positioning MI355X against generationally.
That leaves a real hole for anyone renting hardware in 2026 rather than reading a roadmap slide: H200 is still one of the most heavily deployed inference GPUs in production, it's what Spheron's own catalog runs on the NVIDIA side, and nobody has published what happens when MI355X's attention kernel meets H200's specifically, under matched sequence length, head dimension, and precision, independent of everything else MLPerf bundles in. This post runs FlashAttention-3 against Spheron's own H200 SXM5 numbers and holds them against HipKittens' published, independently reproducible AITER benchmark methodology, rather than treating a system-level percentage, against H200 or B200/B300, as though it settles the kernel question.
Test Setup: MI355X and H200, Same Attention Op, Same Config
Software Stack: HipKittens/AITER on ROCm 7.0 vs FlashAttention-3 on CUDA
On the AMD side, HipKittens' benchmarks run on ROCm 7.0, with 500 warmup runs and 100 measurement runs per configuration on random inputs, the exact protocol its authors describe in the paper. That's the methodology a reader would need to match to reproduce AMD's numbers on rented MI355X hardware. Spheron's own catalog is NVIDIA-only today, so the AMD side of this comparison is HipKittens' own published numbers rather than a fresh run on our infrastructure; a reader who wants to reproduce that half needs a provider that actually carries MI355X, such as Oracle OCI, Vultr, TensorWave, or Crusoe.
On the NVIDIA side, FlashAttention-3 runs under CUDA 12.3+ with flash-attn 2.7+, auto-selected by vLLM 0.6+ and SGLang 0.4+ on Hopper hardware. We used the throughput and TFLOPS figures from Spheron's own FlashAttention-3 benchmarking work, run on an H200 SXM5 instance, which carries 141 GB of HBM3e at 4.8 TB/s, the exact spec baseline the FA3 kernel is working against on the NVIDIA side. General provisioning steps for a Spheron GPU instance are covered at Spheron's docs.
Sequence Lengths, Head Dims, Batch Size Held Constant
Where the two sides can be lined up, we held sequence length, head dimension, and precision constant: 2K, 8K, 32K, and 128K sequence lengths, head dim 128 (the standard for Llama, Qwen, and Mistral-family GQA models), causal masking, BF16. That's the configuration space Spheron's own H200 FA3 numbers were measured under. HipKittens' paper doesn't publish a row-by-row breakdown at those exact same sequence lengths against B200, which is precisely the mismatch that makes a clean, single head-to-head table impossible right now. What follows is the best reconciliation of the two datasets that's honest about where they do and don't overlap.
Results: Tokens/Sec and $/Token Head-to-Head
Attention Forward Pass: H200 FlashAttention-3 Tokens Per Second, By Sequence Length
FA3 on H200 SXM5 delivers roughly 260 TFLOPS at 2K sequence length, climbing to about 425 TFLOPS at 8K before easing back to roughly 395 TFLOPS at 32K and 305 TFLOPS at 128K, per Spheron's own FA3 benchmark on H200 and H100. That's the NVIDIA half of this comparison, measured on the exact GPU this post rents.
HipKittens claims its forward attention kernel is "competitive with FlashAttention-3 under comparable settings," but that claim is made against B200, a GPU with 8.0 TB/s of memory bandwidth against the H200's 4.8 TB/s, a 67% bandwidth advantage NVIDIA holds in that specific matchup. Matching FA3-on-B200 is a harder bar to clear than matching FA3-on-H200 would be, since B200's extra bandwidth gives FA3 more headroom to hide memory latency. If MI355X's forward kernel is merely competitive with FA3 running on the faster of NVIDIA's two current chips, the likely, but unconfirmed, implication is that it would at least match, and quite possibly exceed, FA3's H200 numbers directly. That's an inference from the bandwidth gap between B200 and H200, not a measured result, and it's exactly the number that stays unconfirmed until someone runs HipKittens' benchmark script on an actual MI355X-vs-H200 rig with matching sequence lengths.
Attention Backward Pass (GQA)
The backward pass is where the strongest real evidence sits. Independent testing by Wafer, a GPU capacity provider, measured aggregate decode throughput of 952 tokens/sec on an 8-GPU MI355X node against 498 tokens/sec on a 2-node B200 setup running tensor-parallel-16, and 1,568 tokens/sec on an 8-GPU B300 node. Single-stream decode came in at 118, 90, and 172 tokens/sec respectively. None of that is H200, and none of it isolates the attention kernel from the rest of the serving stack, but it's real measured throughput rather than a paper's own self-reported comparison, and the B200 number in particular looks weak partly because of the TP16 communication penalty NVIDIA's setup paid that AMD's single 8-GPU node didn't.
| Configuration | Aggregate decode (tok/s) | Single-stream decode (tok/s) |
|---|---|---|
| MI355X, 8-GPU node | 952 | 118 |
| B200, 2 nodes, TP16 | 498 | 90 |
| B300, 8-GPU node | 1,568 | 172 |
Wafer's independent MI355X vs B200/B300 decode benchmark, July 2026.
Where AMD Wins and Where It Doesn't Yet
GQA Backward: AMD's Real Kernel-Level Edge
Kernel-level, not decode-serving-level, the clearest documented AMD win is the backward pass. As we detailed in our earlier HipKittens kernel-performance coverage, HipKittens outperforms every available baseline, including AMD's own hand-tuned assembly, by 1.2 to 2.4x specifically on GQA backwards, d=64 attention, and other memory-bound kernels. That's not a comparison against FlashAttention-3 on any NVIDIA chip; it's AMD beating its own previous best. But it's meaningful because the backward pass is the operation training and fine-tuning workloads spend the most time in, and it's the case where AMD's software stack struggled hardest before this integration landed inside AITER.
For readers whose interest in MI355X extends past inference serving into actually training on it, our ROCm training guide for AMD GPUs covers what that backward-pass edge looks like in a full training loop rather than an isolated kernel benchmark.
The Prefill Shape-Mismatch Bug That Still Costs AMD Latency
The honest counterweight to the backward-pass win is a real production bug Wafer hit and documented. Running a 172,000-token Kimi K3 request on an 8-GPU MI355X node at tensor-parallel-8, the model produces 12 attention heads per rank, but MI355X's fast attention kernel only accepts 4, 8, or multiples of 16 heads. That mismatch forced a fallback path: the request took 51 seconds against 23 seconds on a comparable B300 node, more than double the latency for the identical prompt. Zero-padding to a compatible head count fixed it, raising throughput from 4,000-7,000 to roughly 13,000 tokens/sec. That's not a fundamental architecture limit, it's a shape-coverage gap in the kernel, but it's the kind of thing that only shows up when you actually run a real workload rather than a benchmark harness tuned to hit clean head-count boundaries, and it's a direct illustration of why ROCm's 90-95% of CUDA throughput on standard PyTorch/vLLM inference doesn't automatically extend to every shape a production model throws at it.
What This Means for Your Next Rental Decision
Cost Per Token at Today's Rates
Start with the hourly gap, because it's the one number that's fully verified on both sides. The only confirmed true on-demand MI355X rate is Oracle OCI at $8.60/GPU-hr. H200 SXM5 on Spheron runs $4.79/hr on-demand and $2.57/hr spot as of 07 Sep 2026, and that Oracle rate is meaningfully higher per GPU-hour than either Spheron figure. Both sides move independently, so run the comparison against live rates rather than a fixed ratio before committing budget.
Wafer's own cost-per-dollar figures, measured at July 2026 spot rates and against B200/B300 rather than H200, tell a different story: roughly 48 tokens/sec per dollar for MI355X against 33 for B300 and just 7 for B200 once the TP16 penalty is factored in. That's a real efficiency win for AMD, but it's a win against NVIDIA's costliest current chips, not against the H200 this post is actually comparing kernel-for-kernel. AMD's kernel-level story getting more credible against B200/B300 doesn't, on its own, make MI355X the cheaper choice against H200's already-lower on-demand hourly rate.
Pricing fluctuates based on GPU availability and provider. The figures above are current as of 07 Sep 2026 and may have changed. Check current GPU pricing → for live NVIDIA rates, and confirm AMD on-demand rates directly with whichever provider you're evaluating.
When It's Still Worth Testing MI355X Yourself
Test MI355X directly if your workload is genuinely memory-capacity-bound: 288 GB of HBM3e comfortably holds models that need multi-GPU sharding on H200's 141 GB, and that memory headroom is real regardless of what any kernel benchmark says. Test it too if your workload leans on the backward pass, fine-tuning or continued pretraining rather than pure inference, since that's exactly where HipKittens' 1.2-2.4x edge over AMD's own assembly baseline actually lives. Hold off if your stack depends on CUDA-only tooling with no ROCm equivalent, or if you need a verified apples-to-apples number before committing budget; that number, for H200 specifically, doesn't exist yet.
Spheron's own catalog is NVIDIA-only, spanning H100, H200, B200, B300, and A100, so it isn't where you'd test the AMD side of this decision. What it's suited for is the H200 baseline: renting an H200 SXM5 on-demand or spot, with per-minute billing and no long-term contract, is a reasonable way to establish your own current NVIDIA numbers before deciding whether MI355X's narrower, and still partly unproven against H200, kernel-level gap is worth chasing on a provider that actually carries it.
The MLPerf percentage everyone quotes is a system score against B200 and B300. The kernel-level number against H200 still doesn't exist publicly, which is exactly why it's worth establishing your own H200 baseline before betting on someone else's AMD number.
Frequently Asked Questions
Not an isolated one. MLPerf Inference v6.0's per-GPU throughput, once normalized across submissions, already puts MI355X within 20% of H200 on LLaMA 2 70B and GPT-OSS 120B at the system level, and AMD's own press materials frame the same submission against B200 and B300 instead. Either way, that MLPerf number bundles GEMM, attention, KV cache paging, and scheduling into one throughput figure. HipKittens' own paper isolates its attention kernel, but only benchmarks it against NVIDIA B200, never H200. This post assembles the closest available kernel-level comparison by holding FlashAttention-3's published H200 throughput against HipKittens' own reproducible AITER benchmark numbers, but a true same-op, same-config, side-by-side MI355X-vs-H200 kernel test still doesn't exist publicly.
HipKittens is a tile-based kernel programming framework for AMD's CDNA3/CDNA4 GPUs, built by Stanford's Hazy Research lab, which shipped as an official AITER backend in March 2026. Its authors describe the forward attention kernel as competitive with FlashAttention-3 under comparable settings, but that comparison was run against NVIDIA's B200, which has roughly 67% more memory bandwidth than the H200 this post benchmarks against.
The only verified true on-demand MI355X rate is Oracle OCI at $8.60/GPU-hr. H200 SXM5 on Spheron runs $4.79/hr on-demand and $2.57/hr spot as of 07 Sep 2026, and that Oracle rate sits well above either Spheron figure. Most cheaper MI355X quotes circulating elsewhere are spot or preemptible, not on-demand, so they aren't a fair comparison against either rate.
Independent testing by Wafer found an 8-GPU MI355X node hitting a cold-prefill bottleneck on a 172,000-token request: at tensor-parallel-8, the model produced 12 attention heads per rank, and MI355X's fast attention kernel only accepts 4, 8, or multiples of 16 heads. That shape mismatch forced a fallback path, taking 51 seconds against 23 seconds on a comparable B300 node. Zero-padding to a compatible head count raised throughput from 4,000-7,000 to roughly 13,000 tok/s.
The backward pass is AMD's strongest documented case. HipKittens outperforms every available baseline, including AMD's own hand-tuned assembly, by 1.2 to 2.4x specifically on GQA backwards, d=64 attention, and other memory-bound kernels. That matters because the backward pass is the workload where AMD's tooling struggled hardest before this integration, and it's the operation most relevant to fine-tuning and training rather than pure inference serving.
