GPU Memory Hierarchy Diagram: Registers to HBM Explained

A GPU memory hierarchy diagram usually shows four boxes stacked from fastest to slowest: registers, shared memory/L1, L2, and HBM, with an arrow pointing down. What most versions of that diagram leave out is any number attached to the boxes, so a reader walks away knowing the shape of the hierarchy without knowing why it matters for a kernel that's actually running on it. This is that same diagram, with the cycle counts from NVIDIA's own tuning guides and independent microbenchmarking papers, followed all the way to why two GPUs with the same peak-TFLOPS spec sheet can post different real throughput on the same kernel.

TL;DR: What the GPU Memory Hierarchy Diagram Shows

• Registers: 256 KB (64K 32-bit) per SM on Ampere, Hopper, and Blackwell, the fastest, fully private tier, per NVIDIA's Ampere tuning guide.
• Shared memory/L1: H100 configures up to 228 KB per SM, 39% more than A100's 164 KB max, per NVIDIA's Hopper tuning guide.
• L2 cache: 40 MB on A100, 50 MB on H100, split into two partitions so cross-partition access takes about 2x longer, per Chips and Cheese.
• HBM: 3.35 TB/s on H100 SXM vs roughly 1.5-2.0 TB/s on A100, below A100's ~19 TB/s on-chip SRAM, per the FlashAttention paper.
• Spheron lists H100 on-demand at $2.65/hr as of 11 Sep 2026, per-minute billing, to test occupancy before a longer rental. Compare H100 GPU pricing.

The GPU Memory Hierarchy Diagram: Four Tiers From Registers to HBM

Every tier in the hierarchy trades speed for capacity in the same direction: the closer a memory sits to the SM's arithmetic units, the faster it is to reach and the less of it exists. Here's the same diagram most GPU architecture decks show, with the H100 numbers from NVIDIA's own tuning guide attached to each box instead of left blank:

Capacity grows by roughly three orders of magnitude at every step down this stack, and latency grows right alongside it. NVIDIA's own tuning guides for Ampere, Hopper, and Blackwell publish the exact per-generation capacities behind each box, and they're a better starting point than a generic diagram because the numbers change from one architecture to the next.

Registers: Per-Thread, Zero Contention, Zero Sharing

Registers are the fastest tier because they aren't shared with anything. Each thread owns its slice of the SM's register file, and no other thread, warp, or block can read or write it. On Ampere, Hopper, and Blackwell alike, that file holds 64K 32-bit registers per SM, 256 KB total, and a single thread can claim at most 255 registers before the compiler has to start making trade-offs, according to NVIDIA's Ampere tuning guide. NVIDIA's Blackwell tuning guide confirms B200 keeps the same 256 KB combined capacity per SM as Hopper, so the register budget hasn't grown even as peak compute has.

That fixed size is the whole reason register allocation is a live design decision for kernel authors rather than something a compiler quietly handles: 256 KB divided across every thread a block wants to run concurrently runs out fast.

Shared Memory / L1: Programmer-Managed SRAM Inside the SM

One step down sits a pool of on-chip SRAM that Ampere, Hopper, and Blackwell all physically unify as L1 cache and shared memory, split by a configurable carveout rather than being two separate pieces of silicon. Shared memory is the programmer-facing half: a kernel author explicitly stages data into it and controls what stays resident, unlike L1, which the hardware manages automatically.

The capacity of this pool has grown every generation. V100 shipped 96 KB per SM; A100's Ampere tuning guide raises the unified L1/shared-memory capacity to 192 KB per SM, a 71% increase, with shared-memory carveout options of 0, 8, 16, 32, 64, 100, 132, or 164 KB, so a kernel can dedicate up to 164 KB of that pool to shared memory specifically. H100 pushes the combined capacity to 256 KB per SM, with up to 228 KB configurable as shared memory, a 39% increase over A100's 164 KB shared-memory maximum, per NVIDIA's Hopper tuning guide. Blackwell's B200 holds that same 256 KB per-SM ceiling rather than extending it further, per the Blackwell tuning guide.

L2 Cache: Shared Across Every SM on the Die

L2 is where the hierarchy stops being private to one SM. Every SM on the die reads and writes the same L2, which makes it useful for data that multiple thread blocks touch, but it also means contention is possible in a way it never is at the register or shared-memory tier. Capacity grew from 40 MB on A100 to 50 MB on H100, alongside an increase in L2-to-SM bandwidth, per the Hopper tuning guide. On the Blackwell side, GB200 pushes L2 capacity to 126 MB, according to the Blackwell tuning guide.

Capacity is only half the story, and the other half is a real trap for anyone reading a spec sheet at face value: Chips and Cheese's H100 teardown found that Hopper's larger L2 is physically split into two partitions, and a cross-partition access takes roughly twice as long as one that stays inside the local partition. More on what that costs in the next section.

HBM: Off-Chip, High Capacity, Comparatively Slow

HBM is the tier most people mean when they say "GPU memory," and it's the only one that's physically off the compute die, connected over a wide interface to a separate stack of memory dies. It holds by far the most capacity of any tier, measured in tens or hundreds of gigabytes rather than kilobytes or megabytes, but that capacity comes at the cost of being the slowest tier in the hierarchy by a wide margin.

H100's HBM3 system supports up to 3 TB/s, a 93% increase over the 1.55 TB/s available on A100-40GB, according to the Hopper tuning guide. That's a real generational jump, and it's still the slowest tier on the chip it ships on, as the next section's numbers show directly. For the deeper generational comparison, including HBM4's arrival on Rubin, our HBM3e vs HBM4 vs HBM4e bandwidth guide covers the system-level bandwidth math this post doesn't. HBM bandwidth also isn't the whole story for multi-GPU work: moving data between GPUs, rather than between a GPU's own tiers, runs over NVLink instead, which our NVLink explainer covers separately from the on-chip hierarchy here.

Latency and Bandwidth at Each Tier, With Real Numbers

Capacity numbers tell you how much data fits at each tier. They don't tell you how long it takes to reach that data, and that's the number that actually determines whether a kernel spends its cycles computing or waiting.

Cycle Counts From Microbenchmarking Papers: A100, H800, and the Generation Before

Two independent microbenchmarking papers measured GPU memory latency with pointer-chase tests, and their numbers aren't directly comparable to each other because they use different stride patterns and cache-bypass methodology. Treat each as its own study rather than blending them into one table.

The Bandwidth Cliff: ~19 TB/s On-Chip SRAM vs 1.5-3.35 TB/s HBM

Latency tells you how long one access takes. Bandwidth tells you how much data you can move once you're streaming, and the gap between on-chip and off-chip memory is even starker there than in the cycle-count tables.

On H100, HBM bandwidth climbs to 3.35 TB/s on the SXM variant, per the Hopper tuning guide, but the on-chip tiers scale too, so the ratio between them doesn't collapse just because the absolute HBM number goes up.

L2's position in this picture is more complicated than "faster than HBM, slower than SRAM." Chips and Cheese's H100 analysis found near-partition L2 read bandwidth exceeding 5.5 TB/s, but the full 50 MB aggregate read bandwidth comes out to 3.8 TB/s, which is actually lower than A100's smaller, unified 40 MB L2 at 4.5 TB/s. Splitting the cache into partitions bought capacity at the cost of aggregate bandwidth, and a far-partition access on H100 carries about as much latency as a discrete GPU's VRAM access on a Radeon RX 6900 XT, per the same analysis. A bigger L2 on a spec sheet is not automatically a faster one.

All of this rolls up into a single number that predicts, before you've profiled anything, whether a given kernel will be memory-bound or compute-bound: the roofline model's ridge point. A kernel with lower arithmetic intensity than the ridge point is memory-bound no matter how fast the GPU's tensor cores are, which is exactly the mechanism behind the peak-TFLOPS-versus-real-throughput gap covered below.

Why Kernel Authors Fight for Shared Memory and Registers

The latency and bandwidth numbers above explain why kernel authors treat registers and shared memory as scarce resources worth fighting over, rather than defaults the compiler can be trusted to handle.

Occupancy: Registers and Shared Memory Are a Shared, Finite Budget Per SM

Occupancy is how many warps (and thread blocks) an SM can run concurrently, and it's capped by whichever resource runs out first: the register file, the shared-memory pool, or the SM's hard limit on resident threads. A kernel that uses more registers per thread leaves less of the fixed 256 KB register file available for other blocks to run alongside it, so fewer blocks fit, and the SM has fewer warps to switch to when one stalls on a memory access. The same trade-off applies to shared memory: a kernel that claims a large carveout for itself narrows how many blocks can co-reside on that SM.

This is a genuinely shared budget, not two independent limits, because the two resources are allocated from the same per-SM pool at the same time.

Register Spilling: The Performance Cliff Nobody Sees on a Spec Sheet

Register spilling is what happens when the compiler can't fit everything a kernel wants in registers at the target occupancy. It doesn't fail to compile; it silently spills the excess into local memory, and local memory physically lives in the same off-chip space as global memory, carrying HBM-level latency instead of register-level latency, per the same NVIDIA forums discussion of shared-memory and occupancy.

Put the two cycle-count tables above next to that fact and the size of the cliff becomes concrete: a variable that should cost roughly 23-40 cycles to touch in shared memory or registers instead costs somewhere around 290-480 cycles, depending on generation and measurement methodology, because it landed in local memory instead. None of that shows up on a spec sheet. It shows up as a kernel that's mysteriously slower than its theoretical occupancy would predict, and the only way to catch it is to check the compiler's own register-usage report or profile with a tool built for exactly this kind of gap.

Tiling: How FlashAttention Uses Shared Memory to Skip the HBM Round-Trip

FlashAttention is the clearest production example of a kernel deliberately restructured around this hierarchy rather than around the arithmetic alone. FlashAttention's core move is tiling: it processes attention in blocks small enough to fit inside on-chip SRAM, using a running softmax to combine tile results correctly, so that full score matrix never has to make the round-trip to HBM at all.

The FLOP count doesn't change. What changes is exactly the thing the tables above quantify: how much traffic crosses the roughly order-of-magnitude bandwidth gap between on-chip SRAM and off-chip HBM. Our FlashAttention explainer covers the full mechanism, the version-by-version changes, and what the resulting memory savings buy in serving capacity.

How This Explains Price-Performance Gaps Between GPUs

Everything above answers a question that a peak-TFLOPS number alone can't: why two GPUs advertising similar compute throughput sometimes deliver noticeably different real throughput on the same kernel.

Same Peak TFLOPS, Different Cache and Register Budgets

Register file size hasn't grown in step with peak compute. Ampere, Hopper, and Blackwell all carry the same 256 KB register file per SM, per NVIDIA's own tuning guides for each generation, even as peak FLOPS climbed substantially across those three architectures. Shared-memory capacity followed a similar pattern: it grew from A100 to H100, then held flat from H100 to B200 at 256 KB combined L1/shared capacity per SM. A kernel that was register-bound or shared-memory-bound on one generation doesn't automatically stop being bound on the next generation just because the peak-TFLOPS line on the spec sheet went up.

L2 behaves the same way in the other direction: a bigger L2 doesn't guarantee more effective bandwidth, as Hopper's partitioned design shows directly. Two GPUs with identical peak TFLOPS and even identical L2 capacity can still post different real throughput on an L2-heavy kernel if one of them has to cross a partition boundary more often than the other. This is the same gap our GPU spec sheet vs real-world performance guide measures from the compute side, using DCGM's tensor-pipe-active metric; this post supplies the memory-hierarchy mechanism, occupancy limits, register pressure, and cache partitioning, that produces part of that gap.

What to Check Before Renting by the Hour

Before renting a GPU by the hour for a specific kernel, four numbers from this hierarchy matter more than the headline TFLOPS figure:

• Register file size and max registers per thread, since that sets the occupancy ceiling for a register-heavy kernel before it ever touches memory.
• Configurable shared-memory capacity per SM, since that's the resource a tiled kernel (like FlashAttention) is designed around.
• L2 capacity and partitioning, since a partitioned L2 can cost real bandwidth even when its total capacity looks larger on paper.
• HBM bandwidth for the specific SKU, since memory-bound kernels are bound by this number regardless of how much compute sits behind it.

Spheron lists on-demand and spot rates across 5+ providers for H100, H200, B200, B300, A100, GH200, L40S, and several other SKUs, with per-minute billing and no minimum commitment. That combination matters specifically for this kind of check: it's cheap to rent an hour on a candidate GPU, profile a real kernel's occupancy and cache behavior against it, and confirm the memory-hierarchy fit before committing to a longer-running instance. What it doesn't do is tell you any of the four numbers above directly; a price-per-hour page reports dollars per GPU-hour, not occupancy or cache-hit rate, so that check still has to run through a profiler.

Pricing fluctuates based on GPU availability. Spheron rates above are live as of 11 Sep 2026; other providers reflect their most recent published rates and may have changed. Check current GPU pricing → for live rates.

A Worked Example: Where a Matmul Kernel Spends Its Time

The numbers above are easiest to internalize by tracing one concrete kernel through all four writable tiers, since a dense matrix multiply is the workload most of this hierarchy was shaped around in the first place.

Tracing One Tile Through Registers, Shared Memory, L2, and HBM

Take a standard tiled FP16 matmul kernel computing a 128x128 output tile, the tile size common to CUTLASS-style GEMM kernels, with a 32-deep reduction dimension per iteration. The kernel's working set moves through the hierarchy in a specific, predictable order:

1. Global loads from HBM. The kernel first pulls the relevant 128x32 and 32x128 slices of the two input matrices from HBM. This is the slowest step per byte moved, which is exactly why it happens once per tile rather than once per output element.
2. Staging through shared memory. Those slices land in shared memory, not directly in registers, because every thread in the block needs to reuse the same input values many times over as it computes its share of the 128x128 output. On A100, that staging buffer competes for a slice of the 164 KB shared-memory maximum available on that SM; on H100, it competes for a slice of the larger 228 KB ceiling. A larger shared-memory budget lets the kernel author pick a bigger tile, or double-buffer the next tile's load while the current one is still computing, both of which reduce how often the kernel has to wait on the HBM round-trip in step 1.
3. Accumulation in registers. Each thread computes its slice of the output tile using values pulled from shared memory, and the running accumulator stays in registers for the full duration of the reduction loop, because registers are the only tier fast enough to update on every single fused-multiply-add without stalling the pipeline. This is also the step where register spilling becomes a live risk: a tile size chosen too aggressively for the accumulator can push register usage per thread past what the 256 KB per-SM file supports at the block count the kernel wants, forcing exactly the local-memory spill described above.
4. L2 as the overflow tier. As different thread blocks across the SM grid touch overlapping regions of the input matrices, L2 catches a meaningful share of those repeated reads before they'd otherwise have to go all the way back to HBM. This is where Hopper's partitioned L2 becomes visible in practice: a block whose L2 traffic happens to land in the local partition sees materially better latency than one whose traffic crosses to the far partition, even though both blocks are running the identical kernel.
5. Global store back to HBM. Once the reduction loop finishes, the finished output tile writes back out to HBM, the one step in this sequence that's unavoidable no matter how well the earlier steps are tuned, because the result has to end up somewhere with enough capacity to hold it.

The tile-size decision in step 2 is the crux of the whole hierarchy in miniature: a bigger tile amortizes the HBM round-trip in step 1 over more useful compute in step 3, but only up to the point where the shared-memory and register budgets in steps 2 and 3 run out. That's the same trade-off FlashAttention makes at the attention-matrix level rather than the matmul-tile level. To see where a real kernel is actually spending its time rather than reasoning about it in the abstract, our GPU monitoring guide covers the Nsight Compute and DCGM metrics that show register usage, shared-memory occupancy limits, and L2 hit rate for a running kernel directly.

Two GPUs with identical peak-TFLOPS numbers can post different real throughput on the same kernel once register file pressure, shared-memory budget, and L2 partitioning enter the picture, so it's worth profiling before committing to a long rental. Spheron lists on-demand and spot GPU rates across 5+ providers with per-minute billing and no minimum commitment, which makes it cheap to rent an hour on a candidate GPU and check its actual occupancy before scaling out.

Get started on Spheron →