HBM3e vs HBM4 vs HBM4e for LLM Inference: GPU Memory Bandwidth Decision Guide (2026)

HBM4 ships on the NVIDIA Vera Rubin in H2 2026, creating a real decision point for teams planning inference infrastructure. Most coverage of this topic is vendor spec sheets. This guide is deployment math: bandwidth ceilings, TPS limits per GPU, and a cost-per-bandwidth comparison across HBM generations so you can pick the right tier without paying for headroom you won't use for 18 months. Check current GPU pricing before making any capacity commitment since spot rates shift weekly.

TL;DR: HBM3e vs HBM4 vs HBM4e for LLM Inference

• Verdict: rent HBM3e in 2026. It tops out at 10 TB/s on NVIDIA B300, while HBM4 reaches up to 22 TB/s on the Rubin GPU, shipping in H2 2026.
• JEDEC standard: JESD270-4 sets HBM4 at up to 8 Gb/s per pin across a 2,048-bit interface, about 2 TB/s per stack, doubling HBM3's independent channel count.
• Vendor parts: Samsung's HBM4 hits 3.3 TB/s per stack and Micron's exceeds 2.8 TB/s per stack, both over 2x HBM3E's per-stack bandwidth, per 2026 announcements; SK hynix showed a 16-layer, 48 GB HBM4 part at CES 2026.
• Spheron pricing: H200 SXM5 (HBM3e, 4.8 TB/s) runs $4.79/hr on-demand on Spheron as of 13 Nov 2026; HBM4 isn't rentable yet. Check live GPU pricing.
• HBM4e: Samsung targets 16 Gbps per pin for HBM4e, over 20% faster than HBM4, with no GPU product announced as of 2026.

What is HBM3e?

HBM3e (High Bandwidth Memory 3 Extended) is the current-generation DRAM standard used on NVIDIA H200, B200, B300, and AMD MI355X GPUs. It is a faster revision of HBM3, raising per-pin data rate from roughly 6.4 Gbps to roughly 9.6 Gbps on a 1,024-bit-per-stack bus. Per-stack bandwidth lands near 1.2 TB/s. Stacks ship in 12-Hi and 16-Hi configurations, which is why H200 (141 GB) and B200 (192 GB) carry different VRAM totals despite using the same memory type.

HBM3e matters for LLM inference because autoregressive decode is memory-bound, not compute-bound. The H100 itself ships in three different form factors with different memory bandwidth numbers for exactly this reason; see our H100 NVL vs SXM5 vs PCIe guide for how that plays out within a single HBM generation. The decode phase reads the full model weight tensor from VRAM once per generated token, and the time that read takes is fixed by memory bandwidth. Moving from H100 (HBM3 at 3.35 TB/s) to H200 (HBM3e at 4.8 TB/s) raises the single-GPU TPS ceiling on a 70B FP16 model from about 24 to about 34 with no other change. Moving to B200 (HBM3e at 8 TB/s) raises it to about 57. The HBM generation is the most direct lever on inference latency at low batch sizes.

The successor, HBM4, doubles the I/O width per stack from 1,024 bits to 2,048 bits and ships on the Vera Rubin in H2 2026. HBM4e has been spec'd by JEDEC but is not announced on any GPU yet. The rest of this guide is the deployment math behind those numbers. For the full generation-by-generation compute and NVLink comparison alongside the memory story, see our Rubin vs Blackwell vs Hopper guide.

Quick Answer: Which HBM Generation for Which Workload

Why Decode Is Memory-Bandwidth-Bound (Not Compute-Bound)

The roofline model describes two performance ceilings for any GPU workload:

Arithmetic Intensity = FLOPs executed / Bytes moved from memory

If your workload's arithmetic intensity falls below the GPU's ridge point (peak FLOPS / peak bandwidth), you are memory-bound. If it falls above, you are compute-bound. That same ridge-point math also plays out inside the GPU, not just at the HBM boundary: registers, shared memory, and L2 each carry their own latency and bandwidth numbers that decide whether a compute-heavy kernel actually reaches its FLOPS ceiling, which our GPU memory hierarchy diagram breaks down cycle by cycle.

For autoregressive decode at batch size 1, the arithmetic intensity is approximately 1 FLOP per byte. The GPU reads all model weights from VRAM once per token step, performs a small number of operations on each value, then moves on. With a 70B FP16 model, that means reading approximately 140 GB of weights per token. The time this takes is fixed by bandwidth, not FLOPS:

These are hard lower bounds. No amount of additional FLOPS can push TPS above the bandwidth ceiling at batch size 1. You can double the H100's FLOPS budget and the 42 ms lower bound stays.

The prefill phase is the opposite: prefill is compute-bound because all input tokens are processed as a batch, creating high arithmetic intensity. Adding FLOPS improves prefill speed. Adding bandwidth improves decode throughput. They are different problems. For the foundational concepts behind why bandwidth, not just capacity, decides throughput, see our dedicated vs shared GPU memory guide.

For the full KV cache analysis and how context length compounds the VRAM pressure problem, see AI's Memory Wall Problem.

HBM Generations: Full Specification Comparison

Each HBM generation is defined by stack height (more layers = more DRAM capacity), per-pin bandwidth (die speed), and I/O width (how many pins per stack). HBM4 breaks the pattern by doubling the I/O width:

The key change from HBM3e to HBM4 is the I/O width doubling from 1,024 to 2,048 bits per stack. Per-stack bandwidth increases from ~1.2 TB/s to ~1.5+ TB/s entirely because of the doubled I/O width. HBM4's per-pin speed (6.4 Gbps/pin) is the same as HBM3, not faster than HBM3e. The bigger driver of the 22 TB/s total on Vera Rubin is more stacks combined with that doubled bus width. This is a structural, not incremental, change, which is why HBM4 represents a generation boundary rather than a speed bump.

On HBM4e: JEDEC published the HBM4e standard as the next iteration after HBM4. As of May 2026, no GPU vendor has announced a product shipping HBM4e. It does not factor into any real deployment decision today, and bandwidth specs are not confirmed by any manufacturer.

One important nuance: HBM3e bandwidth varies significantly across GPU models because B200 and B300 use different configurations. B200 achieves 8 TB/s; B300 achieves 10 TB/s. Do not treat "HBM3e = 8 TB/s" as a fixed fact. Always tie bandwidth numbers to the specific GPU model.

If your workload needs the full 288 GB of HBM3e per GPU at rack scale, Spheron has GB300 NVL72 availability you can reserve right now: fill in your GPU count, timeline, and workload on the form, and the team confirms availability and reaches out within a business day.

Per-GPU Bandwidth Comparison: H100 to Vera Rubin

†Approximate tokens-per-second ceiling for a 70B FP16 model (140 GB weights) at batch size 1, calculated as GPU_bandwidth / model_weight_bytes. This is the hard upper bound for single-request decode throughput and does not account for KV cache reads, attention overhead, or framework buffers.

Note: AMD MI355X and Vera Rubin are included for completeness but are not available on Spheron as of this writing. See GPU rental for Spheron-available options. That 22 TB/s Vera Rubin figure is the per-GPU number; once you're deciding how many Rubin GPUs to actually rent together, the Vera Rubin NVL4 vs NVL72 guide covers how aggregate bandwidth and HBM4 capacity scale across the two available form factors.

Bandwidth-Bound TPS Ceiling: Worked Examples

The formula is straightforward:

Max TPS (memory-bound) = GPU_bandwidth_bytes_per_sec / model_weight_bytes_per_token
model_weight_bytes = parameter_count × bytes_per_param

At batch size 1, the GPU reads all weights once per token step. The TPS ceiling is the bandwidth divided by the weight footprint.

Llama 3.1 70B at FP16 (140 GB weights)

H100 at 80 GB and GH200 at 96 GB cannot hold the 140 GB model at FP16. Both require 2-way tensor parallelism or FP8 quantization, which halves model weights to ~70 GB and fits on a single GPU with room for the KV cache. H200 at 141 GB fits the model weights at FP16, but 140 GB of those 141 GB are occupied by the weights themselves, leaving only ~1 GB for KV cache. At batch size 1 with a 4K context, Llama 3.1 70B's KV cache requires approximately 1.25 GB on its own. Any non-trivial batch size or context length will overflow the available headroom and cause OOM errors. In practice, FP8 quantization is strongly recommended on H200: it halves model weights to ~70 GB, leaving ~71 GB for KV cache and enabling meaningful batch sizes and context lengths. You can now rent AMD MI300X on Spheron: its 192 GB at 5.3 TB/s fits the 140 GB model comfortably in a single card, leaving room for KV cache, though it runs ROCm rather than CUDA and needs its own validation pass. For techniques to reduce KV cache memory pressure and maximize single-GPU utilization, see KV Cache Optimization Guide.

DeepSeek V3 at FP8 (671B MoE, ~37B active parameters per step)

For MoE models, only the active expert weights are read per decode step. DeepSeek V3 activates approximately 37B parameters per token step at FP8 precision, meaning ~37 GB of weights per step.

Note: the full model at FP8 is ~671 GB, requiring at least 9xH100 (9 × 80 GB = 720 GB), 5xH200 (5 × 141 GB = 705 GB), or 4xB200 (4 × 192 GB = 768 GB) for single-instance serving. The TPS ceiling below is calculated per individual GPU (per-GPU bandwidth / active weight bytes). In a multi-GPU setup, aggregate system TPS scales roughly with GPU count, minus interconnect overhead.

This is an approximation for the bandwidth-ceiling calculation only. The ~37B active parameter count is DeepSeek V3's nominal per-step activation, but actual performance depends on routing patterns, sparsity, and all-to-all communication latency in multi-GPU setups.

Large dense model proxy: 405B at FP8 (~405 GB weights)

At FP8, a 405B model weighs approximately 405 GB. No single GPU currently has sufficient VRAM to hold it. Even the B300 and Vera Rubin at 288 GB each fall short. The minimum viable configurations are 2xB300 (2 × 288 GB = 576 GB) or 2x Rubin GPU (2 × 288 GB = 576 GB).

Only B300 and Vera Rubin can serve this model in a 2-GPU configuration (2 × 288 GB = 576 GB ≥ 405 GB). With 2-way tensor parallelism, the effective per-step bandwidth is the sum of both GPUs' bandwidth:

Actual throughput is below these ceilings due to KV cache reads, attention, NVLink communication overhead, and framework overhead. For the full profiling methodology, see AI's Memory Wall Problem.

When FLOPs Matter: Prefill vs Decode

Bandwidth is the only lever for decode at small batch sizes. But as batch size grows, the workload shifts:

Prefill is compute-bound. During prefill, the model processes all input tokens at once, giving high arithmetic intensity. Adding FLOPS directly reduces time-to-first-token. The B200's 2,250 FP16 TFLOPS (vs 1,979 on H100) and 4,500 FP8 TFLOPS translate directly into faster prefill throughput.

Decode at batch 1-4 is bandwidth-bound. Only bandwidth matters here. H200's 43% bandwidth advantage over H100 translates to 43% more tokens per second on memory-bound decode.

Decode at batch 64+ starts to shift. As more requests are batched together, the same weight bytes are shared across more tokens, increasing arithmetic intensity. For batch sizes above the roofline crossover point in the table above, FLOPS and bandwidth both matter.

For a real-time chatbot serving batch 1-8, bandwidth is the only hardware decision that matters. For a high-throughput API endpoint batching 64+ requests, you need to think about both. For teams running separate prefill and decode fleets, see Prefill-Decode Disaggregation on GPU Cloud for the infrastructure patterns.

Quantization as a Bandwidth Multiplier

Quantization reduces bytes per parameter, which directly scales the TPS ceiling. On a fixed GPU, lower precision is a free bandwidth upgrade:

A few clarifications:

FP4 tensor core hardware support is only available on Blackwell (B200/B300). On H100/H200, FP4 has no hardware backing and falls back to FP8 or slower emulation. INT4 via AWQ or GPTQ works on H100/H200 but uses software dequantization, so the effective bandwidth benefit is smaller than the theoretical 4x.

AMD's MXFP4 format is available on MI355X and uses a different numerical standard than NVIDIA's NV-FP4. The byte width is the same (0.5 bytes/param), but model accuracy and kernel availability differ by framework.

For framework setup and accuracy tradeoffs on FP4, see FP4 Quantization on Blackwell GPU Cloud. For INT4/INT8 workflows with AWQ, see AWQ Quantization Guide for LLM Deployment.

HBM Pricing and Bandwidth Per Dollar

As a worked example at the Spheron rates recorded on 15 May 2026, frozen so the per-dollar ratios stay consistent:

Spheron's live rates, to rerun the ratios today: H100 $2.39/hr on-demand and $2.20/hr spot; H200 $4.79/hr and $3.39/hr; B200 $10.17/hr and $6.01/hr; B300 $11.47/hr and $5.87/hr.

Pricing fluctuates based on GPU availability. Spheron rates above are live as of 13 Nov 2026; other providers reflect their most recent published rates and may have changed. Check current GPU pricing → for live rates.

At the 15 May 2026 rates, on spot pricing, H200 delivered the best bandwidth-per-dollar at 2.54 TB/s per $/hr, ahead of B200 (2.12) and H100 (1.98).

On on-demand pricing at those rates, B200 led at 1.09 TB/s per $/hr, with H200 close at 1.02 and H100 trailing at 0.84. The gap between H200 and B200 on on-demand is small, but B200 also brings native FP4 Tensor Core support and 2.4x the raw bandwidth, which matters for high-concurrency workloads.

Vera Rubin at estimated $8-12/hr would deliver 22 TB/s, giving 1.8-2.75 TB/s per $/hr depending on final pricing. At the high end of pricing estimates, H200 still matches or beats Vera Rubin on bandwidth-per-dollar. HBM4's advantage over HBM3e on this metric is not guaranteed until cloud spot pricing normalizes, which historically takes 6-12 months after a new generation launches.

Decision Matrix: Which HBM Generation to Rent on Spheron

Migration Path: HBM3 to HBM4

HBM3 (H100) to HBM3e (H200): Drop-in swap. Same Hopper architecture, same CUDA code, same TensorRT-LLM and vLLM configs. Only change: your batch size can increase ~43% for the same latency target because bandwidth increased 43%. No code changes, no new dependencies, no new precision formats to configure.

HBM3e (H200) to HBM3e (B200): Architecture change from Hopper to Blackwell. For FP8 and FP16 workloads, existing vLLM configs run without modification. To unlock native FP4 Tensor Cores, you need TensorRT-LLM 0.17 or later and an FP4-quantized checkpoint. The B200's 8 TB/s bandwidth means the same 70B model now has ~57 TPS ceiling vs ~34 TPS on H200 at FP16, and ~137+ TPS at FP4. For the H100-to-B200 migration math and quantization setup, see FP4 Quantization on Blackwell GPU Cloud.

HBM3e (B200) to HBM4 (Vera Rubin): H2 2026 cloud availability from the first confirmed cohort (AWS, Google Cloud, Azure, CoreWeave). Architecture change from Blackwell to Rubin. New NVLink 6 at 3.6 TB/s per GPU for multi-GPU configurations. Software stack compatibility with TensorRT-LLM and vLLM is expected to follow within 2-4 months of GA availability, based on the pattern from Blackwell. Broader cloud availability including Spheron is expected in 2027.

For a broader workload-based decision framework beyond just memory bandwidth, see best GPU for AI inference in 2026.

Spheron aggregates H100, H200, B200, and B300 instances from data center partners globally - pick the HBM generation that matches your workload's bandwidth requirements without paying for Vera Rubin-tier headroom you don't need today.

Spheron H200 → | B200 SXM6 on Spheron → | View all GPU pricing →