How Much GPU Memory Do I Need? 2026 Sizing Guide (7B-400B)

How much GPU memory do I need to run the model you just found on HuggingFace? Here's the short version: match the model to a row in the table below and rent the cheapest card that clears it. This post skips the byte-by-byte derivation on purpose (our GPU memory requirements guide already owns that math in full) and goes straight from model size to GPU tier to an hourly price you can actually rent today.

Quick Answer: How Much GPU Memory Do I Need?

Your VRAM number comes down to three inputs: parameter count, the precision you serve at, and how much your context length and concurrency add on top for KV cache. Rather than reproduce that formula here, this table starts from the answer that actually matters when you're renting hardware, which card clears which model tier and what it costs. It's Spheron's catalog sorted cheapest to priciest, pulled from Spheron's live GPU marketplace, which aggregates on-demand and spot capacity across 5+ providers:

Pricing fluctuates based on GPU availability. The prices above are based on 30 Aug 2026 and may have changed. Check current GPU pricing → for live rates.

Two things trip people up here. First, VRAM tracks total parameters, not "active" parameters on a Mixture-of-Experts model, we'll get to why below. Second, this table assumes short context and light concurrency: push the context window to 128K or serve a handful of requests at once and the KV cache alone can add tens of gigabytes on top of these numbers. The section right below breaks that "fits" column down model by model; for the full derivation, the exact KV cache formula, and training memory math, keep our GPU memory requirements for LLMs guide open in another tab while this page stays open for the buying decision.

Why VRAM Is the Real Cost Driver in LLM Deployment

An H100 has roughly a quadrillion FLOP/s of peak compute, yet an 8B model running on it generates only about 200 tokens per second, because single-request decoding is bound by memory bandwidth, not by how much math the GPU can do. That's the core reason VRAM decides your GPU bill more than raw compute does: you're not paying for FLOPs you'll never use, you're paying for enough HBM to hold the model and enough bandwidth to stream it fast. We go deeper on why more GPUs don't fix a bandwidth-bound workload in that memory wall guide.

Underestimate VRAM and one of two things happens: the process crashes on load with an out-of-memory error, or it technically runs but spills into CPU memory and drops throughput by an order of magnitude. Overestimate it and you're renting an H200 for a model that would have run fine on an L40S at a third of the hourly rate. Getting the number right the first time is the difference between a clean deployment and either an outage or a wasted budget line.

How Much GPU Memory Do I Need by Model Size: 7B to 400B+

7B-8B Models

The entry tier. Llama 3.1 8B, Mistral 7B, and Qwen 3 8B all land in the same bracket: small enough that a single RTX 4090 (24 GB) covers any precision from FP16 down to INT4 with room to spare, and it's the cheapest on-demand card in Spheron's catalog to boot. There's rarely a reason to quantize a model this size unless you're trying to free up headroom for a bigger KV cache under heavy concurrency.

13B-34B Models

Phi-4 14B, Qwen 3 32B, and similar mid-size models are where quantization starts earning its keep. At FP16 you need an RTX 5090 (32 GB) or larger; drop to INT4 and the same model fits back on an RTX 4090 with room left for KV cache. Full byte-level figures for each precision are in the VRAM math guide if you need them for a capacity plan.

70B-72B Models: Llama 70B VRAM Requirements

This is the tier most teams actually ask about, and Llama 3.1 70B is the model that defines it. It has 70B parameters, a 128K-token context window, and uses grouped-query attention (GQA) rather than standard multi-head attention, per Meta's Llama 3.1 model card, hosted on HuggingFace. GQA matters here because it's the reason a 70B model is practical to serve on affordable hardware at all: cutting KV heads down from a full multi-head configuration to a handful of shared heads shrinks the per-token cache by roughly 8x compared to standard attention on a model this size.

What that buys you on Spheron's catalog: INT4 gets Llama 70B onto a single 48 GB L40S, INT8 needs an 80 GB A100 or H100, and FP16 needs a 141 GB H200 or a pair of 80 GB cards split with tensor parallelism. If you want the underlying byte math (weights, KV cache, and overhead broken out separately), that lives in the full VRAM derivation guide; this post cares about which card you'd actually rent.

Here's why the "fits on one card" answer isn't fixed, though. Say you're serving Llama 3.1 70B at INT4 with a 32K-token context window at batch size 4. Each token in this model's KV cache costs about 0.31 MB at BF16, so four concurrent requests at 32,768 tokens each add roughly 4 x 32,768 x 0.31 MB, about 39 GB of cache on top of the INT4 weights. That's enough to push the real footprint past a single L40S and into A100 or H100 territory, even though the base INT4 weights alone would have fit comfortably. Context length and concurrency, not just model size, decide whether you need to size up.

109B-180B MoE Models

This bucket mixes architectures, and that's the point worth flagging. Llama 4 Scout is a Mixture-of-Experts model (109B total, 17B active) that breaks the intuition that VRAM scales with "how smart the model feels": sizing runs on total parameters, because every expert's weights have to sit resident in memory even though only a fraction fire on any given token. Active parameters affect compute speed, not memory footprint. On Spheron's catalog, Scout's INT4 build fits a single 80 GB H100; FP16 needs multiple cards. Mistral Large 2, a dense 123B model at the top of this bucket, follows ordinary parameter-count math with no MoE adjustment needed. Our MoE inference guide covers expert parallelism and the routing decisions that come with serving MoE architectures in production, and the Llama 4 deployment walkthrough has the vLLM commands if you want to stand one up.

400B+ Models

Llama 4 Maverick (400B total, 17B active) is the current shape of the largest tier most teams will actually deploy. INT4 is the practical entry point on Spheron: it fits on 4x 80 GB H100s or a single B300 (288 GB); FP16 pushes you to 8x H200 or a multi-node setup. Past this size you're firmly in multi-GPU or multi-node territory regardless of quantization, which is the subject of the next section.

Quantization's Impact on VRAM: FP16 vs INT8 vs 4-bit

Quantization is the single highest-leverage lever for fitting a model into less VRAM, because it changes how many bytes each parameter costs to store, not how many parameters exist. What matters for a buying decision isn't the bytes-per-parameter table (that's in the VRAM math guide), it's how many GPU tiers quantization buys you. For Llama 70B specifically:

\*Prices as of 30 Aug 2026; check current GPU pricing → for live rates.

Going from FP16 to INT4 on this model is the difference between a $4.79/hr H200 and a $0.96/hr L40S, roughly a 5x cost cut for the same model. Modern quantization methods make that trade worth taking in most cases: AWQ (Activation-aware Weight Quantization) is a post-training, weight-only method that identifies the roughly 1% of weight channels that matter most based on activation patterns and protects those specifically, rather than quantizing everything uniformly. It does this without backpropagation or reconstruction, which is part of why it generalizes better across domains than earlier reconstruction-based approaches, per the original AWQ paper. For the full mechanics and a step-by-step quantization walkthrough, see our AWQ quantization guide.

Not for every workload, though. AWQ and GPTQ hold up well on straightforward chat, classification, and retrieval tasks. Where they start to slip is multi-step reasoning and math-heavy chains, exactly the workloads teams most often ship without re-testing after quantizing. Run your own eval on that kind of pipeline before treating INT4 as a drop-in replacement for FP16.

One frequent point of confusion: quantizing the model weights doesn't automatically quantize the KV cache. Those are separate levers. If your bottleneck is cache growth at long context rather than weight size, KV cache quantization (FP8 or lower) is the more targeted fix, covered in our KV cache optimization guide.

Single-GPU vs Multi-GPU: When You Need to Shard

The honest answer is: you need multiple GPUs the moment your model's weights, active KV cache, and framework overhead together exceed what one GPU's HBM holds at your real context length and concurrency target, not before. Splitting a model that fits on one card across two just adds inter-GPU communication overhead for no benefit.

A quick gut check on where that line falls, at BF16 with moderate concurrency:

• Fits on one 80 GB GPU (H100 or A100): dense models up to roughly 32B, and 70B models at INT8 or INT4.
• Fits on one 141 GB H200: 70B models at FP16 with modest context, or INT4 models with a large KV cache budget.
• Needs 2+ GPUs: 70B at FP16 with long context or several concurrent requests, and any MoE model past roughly 150B total parameters, regardless of quantization.

Before PagedAttention, LLM serving systems commonly wasted 60-80% of allocated KV-cache memory to fragmentation and over-reservation. "We find that existing systems waste 60%-80% of memory due to fragmentation and over-reservation," the vLLM team wrote when introducing PagedAttention, and their approach cuts that waste to under 4%, which is effectively 2-4x more concurrent requests on the same hardware before you'd otherwise need to add a GPU. That's often a cheaper first move than sharding.

Once you do need to shard, the choice is tensor parallelism (splitting individual layers across GPUs, needs NVLink-class bandwidth, stays inside a node) or pipeline parallelism (splitting whole layers across GPUs, tolerates slower interconnects, scales across nodes). Our multi-GPU vs single-GPU sizing guide covers the cost trade-offs between the two and when NVLink actually earns its premium over InfiniBand or Ethernet.

Matching VRAM Needs to GPU Cloud Pricing Tiers

The pattern holds across every row in the table above: figure out your VRAM number first, then work down the catalog until you find the cheapest card that clears it with headroom. Renting an H200 for a 13B model wastes money; trying to squeeze a long-context 70B model onto an L40S wastes your afternoon debugging out-of-memory errors.

Two adjustments are worth making before you commit to a tier:

Spot capacity cuts the bill on every card that offers it, and the discount grows as the card gets more expensive, roughly 17% off on A100, over 30% off on H200. That's a real saving for batch inference or an autoscaled fleet that tolerates a node disappearing mid-request. It's a worse fit for a single-instance production endpoint with a latency SLA, since spot capacity can be reclaimed without notice. B300 is spot-only in the current catalog for exactly this reason, it's priced and provisioned as burst capacity, not a guaranteed on-demand seat.

Price per GB of VRAM also isn't flat across the catalog. Moving from an L40S to an H100 roughly doubles what you pay per GB, because you're also paying for HBM bandwidth and a newer compute generation, not just capacity. If your actual bottleneck is bandwidth rather than raw capacity, that premium is often worth it: it's the difference between an 8B model doing 200 tokens/second on an H100 versus struggling to keep up on cheaper, slower memory.

Spheron's GPU Catalog Mapped to Common Model Sizes

Same catalog, read the other direction: start from the card instead of the model, useful once you already know which GPU you want to rent.

The 24-48 GB tier (RTX 4090, RTX 5090, L40S) covers almost every 7B-34B open model at FP16 and stretches to 70-72B at INT4 with vLLM or TensorRT-LLM handling the quantized weights directly. It's also where most single-developer and small-team inference workloads actually live, since a lot of production use cases never need more than a 7B-13B model.

The 80-141 GB tier (A100, H100, H200) is the workhorse range for 70B-class models and the smaller MoE architectures. INT8 gets a 70B model onto an 80 GB card with room to spare, and the H200's extra headroom is what lets that same model run at FP16 or serve long context without sharding across two GPUs.

Past 192 GB (B200, B300) you're buying capacity for MoE models in the hundreds of billions of total parameters, or for multi-GPU nodes running FP4-native serving. Check current B300 spot availability against on-demand B200 pricing before committing if reliability matters more to your deployment than raw cost.

For anything not covered above, whether that's a specific fine-tuned checkpoint or a brand-new model release, our LLM VRAM calculator reads the parameter count straight from any HuggingFace model card and matches it to the cheapest GPU with live pricing. If your actual need is fine-tuning rather than inference, the VRAM math is a different, much larger number: see our GPU VRAM requirements to fine-tune LLMs guide for that side of the equation. And if you know your VRAM budget but not which model to run on it, the best open-source LLMs by VRAM tier guide answers the inverse question.

Once you know your VRAM number, matching it to hardware takes minutes on Spheron: bare-metal GPUs from RTX 4090 up to B300, per-minute billing, no contracts.

Explore GPU options on Spheron →