GPU sizing for LLM serving usually starts in the wrong place: model weights. An H100 80GB comfortably holds a 70B model's weights at INT4, and a team reading only the spec sheet will assume that settles the question. It doesn't. The number that actually decides how many people that GPU can serve at once is the KV cache, and it grows with every concurrent request and every token of context in a way weights never do. This post fixes the GPU at one H100, varies context length and batch size, and works out exactly where KV cache takes over as the limiting resource.
TL;DR: GPU Sizing for LLM Serving on One H100
GPU sizing for LLM serving on an H100 80GB is a KV cache budget problem, not a weights-fit problem.
- Fixed cost: model weights don't scale with traffic. An 8B model at FP16 uses about 16GB no matter the user count.
- Variable cost: KV cache scales with every concurrent user and every context token.
- Crossover: on one H100, Llama 3.1 8B at FP16 holds roughly 107 concurrent users at 4K context but only about 3 at 128K.
- Decision rule: quantize the KV cache and cap context length before adding hardware. Spheron's H100 runs $2.65/hr on-demand as of 20 Sep 2026. Compare GPU pricing.
The Math: Model Weights vs KV Cache Growth Per Concurrent User
Two numbers compete for the same pool of VRAM on any inference GPU, and they behave completely differently.
Model weights are a one-time load cost. An 8B parameter model at FP16 (2 bytes per parameter) takes about 16GB regardless of how many requests hit it. A 70B model at FP16 takes roughly 140GB, which is already more than an H100 has, forcing either quantization or a second GPU before you've served a single user.
KV cache is different. During autoregressive generation, the model caches the key and value vectors for every token it has already processed, for every request currently in flight, so it doesn't recompute them on each new token. That cache grows along five axes at once: transformer layers, KV heads, head dimension, sequence length, and concurrent batch size. Layers, heads, and head dimension are fixed by the model architecture, but sequence length and batch size are set by your traffic, which is exactly why the same GPU can serve 100 short-context users comfortably and fall over at three long-context ones.
Grouped-query attention (GQA) is the architectural lever that makes this tractable at all: instead of giving every attention head its own key-value pair (multi-head attention, or MHA), GQA shares KV pairs across groups of query heads, per the paper that introduced the technique. Llama 3.1 continues to use GQA across its model family, as Meta's Llama 3.1 release confirms. The reduction depends on the query-head count, not the KV-head count alone: Llama 3.1 8B pairs 8 KV heads with 32 query heads for a 4x smaller cache than full multi-head attention, while Llama 3.1 70B pairs the same 8 KV heads with 64 query heads for a full 8x reduction, both before you touch quantization at all. Our KV cache optimization guide covers GQA and every other cache-shrinking lever in depth; this post uses the resulting per-token number for one purpose only, answering how many concurrent users a fixed H100 holds at a fixed context length.
This is why the two costs need to be sized separately. A model that fits fine at rest can still be impossible to serve at your target concurrency, and a model that looks memory-hungry on the spec sheet can serve more users than expected if its architecture is KV-cache-friendly. Our GPU memory sizing guide covers the weights side of this in depth, model size to GPU tier with no concurrency assumptions; this post picks up from there and works out what happens once real traffic hits the card.
GPU Sizing for LLM Serving: The KV Cache Formula on H100 80GB
The formula for KV cache memory, in bytes, is:
KV_bytes = 2 x L x H_kv x D x S x B x bytes_per_elementWhere L is transformer layers, H_kv is KV heads (after GQA), D is head dimension, S is sequence length in tokens, B is concurrent batch size, and bytes_per_element is 2 for BF16/FP16, 1 for FP8, or 0.5 for FP4/NVFP4. This section applies that formula specifically to what one H100 80GB can hold at fixed concurrency targets, rather than to the quantization techniques that shrink the per-token number in the first place.
Two models anchor the worked examples below, both using Llama 3.1's GQA configuration (8 KV heads, 128 head dimension):
| Model | Layers | KV heads | Head dim | KV cache per token (BF16) |
|---|---|---|---|---|
| Llama 3.1 8B | 32 | 8 | 128 | ~0.13MB |
| Llama 3.1 70B | 80 | 8 | 128 | ~0.32MB |
Before running the numbers, one practical adjustment: vLLM's default --gpu-memory-utilization setting reserves a large majority of total VRAM, commonly around 90%, for weights plus KV cache combined, leaving the remainder for activation memory and CUDA context overhead. On an 80GB H100, that's a usable pool of about 72GB, not the full 80GB, and every calculation below starts from that number.
It's also worth being clear about what this ceiling isn't. vLLM's max_num_seqs is an upper limit on concurrency, not the actual number served; real concurrency is determined dynamically by available KV cache blocks, according to vLLM's optimization guide. Set max_num_seqs past what your KV cache budget can hold and requests don't run faster, they queue or get preempted. The vLLM documentation describes exactly this failure mode: "Sequence group 0 is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space."
Worked Example: How Many Concurrent Users Fit on One H100 at 4K, 32K, and 128K Context
Here's exactly how the 72GB usable pool splits between weights and KV cache for three configurations, and what each one holds at three context lengths.
Llama 3.1 8B at FP16 (weights: 8B params x 2 bytes = ~16GB, leaving ~56GB for KV cache):
- 4,096 tokens: each user's KV cache costs ~0.52GB (0.13MB x 4,096). 56GB / 0.52GB ≈ 107 concurrent users.
- 32,768 tokens: each user's KV cache costs ~4.2GB. 56GB / 4.2GB ≈ 13 concurrent users.
- 131,072 tokens (128K): each user's KV cache costs ~16.8GB. 56GB / 16.8GB ≈ 3 concurrent users.
Llama 3.1 8B at FP8 (weights: ~8GB, leaving ~64GB for KV cache):
- 4,096 tokens: ~64GB / 0.52GB ≈ 122 concurrent users.
- 32,768 tokens: ~64GB / 4.2GB ≈ 15 concurrent users.
- 131,072 tokens: ~64GB / 16.8GB ≈ 3 concurrent users.
Llama 3.1 70B at INT4 (weights: 70B params x 0.5 bytes = ~35GB, leaving ~37GB for KV cache):
- 4,096 tokens: each user's KV cache costs ~1.3GB (0.32MB x 4,096). 37GB / 1.3GB ≈ 28 concurrent users.
- 32,768 tokens: each user's KV cache costs ~10.5GB. 37GB / 10.5GB ≈ 3 concurrent users.
- 131,072 tokens: each user's KV cache costs ~42GB, which is larger than the entire 37GB KV budget. The model can't serve a single 128K-context request on one H100 at this precision, let alone concurrently.
The pattern holds across every row: weights are the same regardless of how many users connect, and KV cache is what actually runs out. The 70B model at 128K context isn't failing because the GPU is too small for the model, it's failing because one long-context request's cache alone exceeds what's left after weights. That's also exactly the case KV cache quantization exists to fix; FP8 KV cache roughly halves each of the per-user numbers above, which is often the difference between a workload fitting on one card and needing two. For a single moderate-length request, an H200's 141GB changes this math directly, since our H200 deployment guide shows the same Llama 3.1 70B FP16 KV cache arithmetic (roughly 320KB per token) landing comfortably inside the larger memory pool where it doesn't fit an H100 at all.
None of this happens for free at the systems level, either. The 72GB-minus-weights arithmetic above assumes PagedAttention-style on-demand block allocation is already doing its job; without it, every number in this section would be considerably worse. Our guide to continuous batching and PagedAttention on H100 covers how that scheduling actually gets realized in production, including chunked prefill for mixed short- and long-context traffic.
When You Need a Second GPU vs a Smaller Model
Once you know your target concurrent user count and context length, there's a clear order of operations before reaching for a second card.
First, cut the KV cache cost per user. Anyscale's own sizing guidance is blunt about this: halving max_model_len from 8,192 to 4,096 tokens doubles the concurrency a GPU can support, because KV cache reservation per request is cut in half, according to Anyscale's GPU guidance for LLM serving. If your product doesn't need the model's full context window, capping it is the cheapest capacity increase available and costs nothing in hardware.
Second, quantize what's left. FP8 KV cache halves the per-token cost again on Hopper and newer hardware, and NVFP4 on Blackwell halves it again from there. If your traffic has repeated system prompts, RAG context, or shared conversation prefixes, sharing that cached KV state across requests instead of recomputing it from scratch is a separate lever entirely, and it stacks with quantization rather than replacing it.
Third, check whether the model itself has a friendlier architecture. The worked examples above use GQA, which already cuts the cache 4x for Llama 3.1 8B and 8x for Llama 3.1 70B versus full multi-head attention. Models built on Multi-Head Latent Attention compress KV cache further still, cutting it by roughly 98% versus standard multi-head attention for typical configurations. If your use case can tolerate a different model family, that architectural choice can outperform any amount of GPU shopping.
Only then does a second GPU make sense, and even that decision has two shapes: scaling out (another H100 for more total KV cache headroom across the fleet) or scaling up (an H200 with 141GB, which is 1.76x an H100's capacity on the same Hopper compute). Which is cheaper depends on your actual gap. If you're short by a modest amount of KV cache headroom at moderate context, a single H200 often closes that gap without doubling your GPU count; if you need genuinely more raw throughput at a fixed context length, adding H100s to the fleet scales linearly. Either way, check live rates before committing: Spheron lists H100 at $2.65/hr on-demand and H200 at $4.80/hr, as of 20 Sep 2026, aggregated across 5+ providers with per-minute billing, which makes it practical to rent one of each and benchmark your actual workload before locking in a tier. Spheron doesn't set or tune the --max-num-seqs or --gpu-memory-utilization flags that determine your realized concurrency, that configuration work stays on your own serving stack; what changes is which GPU tier you're running that stack on, and how quickly you can switch it if the math above says you sized it wrong.
Hardware generation matters here too, separately from raw VRAM. The VRAM math above is necessary, not sufficient: a newer architecture can clear more concurrent users out of a comparable memory budget through better kernels and higher compute throughput, on top of whatever headroom the memory pool itself provides.
Whatever tier you land on, the concurrency numbers above assume the requests are actually getting served inside a usable latency budget. VMware's own sizing guidance for production LLM inference recommends targeting time-to-first-token under 200 milliseconds and 30 or more tokens per second per user, per VMware Cloud Foundation's inference sizing and performance guide, which is roughly a 33-millisecond inter-token gap. A GPU can technically hold more concurrent sequences in KV cache than it can serve at that latency target, so treat the KV-cache-derived numbers above as a ceiling, not a target: the real number you should provision for is whichever is lower, the KV cache limit or the concurrency at which your P95 latency crosses your SLA.
Quick Reference: Concurrent User Capacity by Model Size and Context Length
Approximate concurrent users on one H100 80GB (72GB usable pool after vLLM's default 90% memory utilization setting), by model, precision, and context length:
| Model and precision | Weights | KV budget | 4K context | 32K context | 128K context |
|---|---|---|---|---|---|
| Llama 3.1 8B, FP16 | ~16GB | ~56GB | ~107 users | ~13 users | ~3 users |
| Llama 3.1 8B, FP8 | ~8GB | ~64GB | ~122 users | ~15 users | ~3 users |
| Llama 3.1 70B, INT4 | ~35GB | ~37GB | ~28 users | ~3 users | 0 (doesn't fit) |
These figures are illustrative, derived from the formula above with vLLM's default memory settings, not a benchmark of measured throughput. Real deployments also depend on request arrival patterns, average output length, and whether continuous batching is keeping the GPU saturated between requests. Use the table to size your starting point, then validate against your own traffic before committing to a fleet size.
Pricing fluctuates based on GPU availability. Spheron rates above are live as of 20 Sep 2026; other providers reflect their most recent published rates and may have changed. Check current GPU pricing → for live rates.
The core takeaway holds regardless of which row you land on: size the GPU to your KV cache budget at your real context length and concurrency target, not to whether the model's weights fit. Weights are the easy number. KV cache is the one that actually decides how many users you can serve.
Once you've run this math against your own model and traffic, the fastest way to confirm it is to rent the GPU and measure it directly rather than trust the arithmetic alone.
Frequently Asked Questions
It depends entirely on model size, precision, and context length, not on the H100 alone. A Llama 3.1 8B model at FP16 leaves roughly 56GB of the H100's 80GB for KV cache after weights and vLLM's framework overhead, which works out to roughly 107 concurrent users at a 4,096-token context, dropping to roughly 13 at 32,768 tokens and roughly 3 at 128,000 tokens. A 70B model at INT4 fits far fewer: about 28 users at 4K context and effectively zero at 128K, because the KV cache alone exceeds the remaining VRAM. Run the formula in this post against your own model and context length rather than using either number directly.
KV cache bytes per token equals 2 x layers x KV heads x head dimension x bytes per element (2 for BF16/FP16, 1 for FP8, 0.5 for FP4/NVFP4). Multiply that per-token figure by context length and by concurrent batch size to get total KV cache demand, then check it against whatever VRAM is left after model weights and vLLM's non-KV overhead. The KV cache optimization guide covers the full derivation and every quantization lever that shrinks the per-token number.
Model weights are a fixed cost that does not change with traffic. KV cache multiplies by context length and by every concurrent request, so it grows unbounded while weights stay flat. At a 128,000-token context, a single Llama 3.1 70B request's KV cache with grouped-query attention runs to tens of gigabytes, and the model's own weights become the smaller line item on the VRAM bill once you're serving more than one or two concurrent long-context users.
Add a GPU when your traffic genuinely needs the context length and concurrency you're serving today and you've already applied the cheap levers: FP8 or NVFP4 KV cache quantization, a context length ceiling matched to real usage rather than the model's max, and prefix caching for repeated system prompts. If those still leave you short, scaling out (a second H100) or scaling up (an H200 with 141GB) both work; which one is cheaper depends on the live per-GPU rate against how much extra KV cache headroom you actually need, which is worth checking against current pricing before committing to either path.
No. max_num_seqs is an upper limit on how many sequences vLLM's scheduler will batch, not a guarantee of that many concurrent users. Real concurrency is determined dynamically by how many KV cache blocks are actually free. Setting max_num_seqs above what your KV cache budget can hold just means requests queue or get preempted once the cache fills, rather than running at the configured ceiling.
