GPU Cluster for AI: Multi-GPU vs Single Large GPU (2026)

Sizing a GPU cluster for AI comes down to one question most teams answer backwards: does the model actually force you onto multiple GPUs, or are you buying a cluster because that's what the last benchmark post recommended? A 70B-parameter model at BF16 needs about 140GB of VRAM for weights alone, more than a single 80GB H100 or A100 can hold, so the decision is made for you. A 13B model at INT4 fits in under 10GB, and bolting on a second GPU does nothing but add network overhead and cost. This is a VRAM-first framework: work out what your model and traffic actually require, map that onto tensor parallelism, pipeline parallelism, and the NVLink or InfiniBand fabric connecting the GPUs, and only then price out the hardware.

For the underlying memory math this guide builds on, see our GPU memory requirements for LLMs guide, which breaks down weights, KV cache, and quantization tier by tier.

When a Single GPU's VRAM Is the Real Bottleneck (and When It Isn't)

A model needs multiple GPUs only when its weights, active KV cache, and framework overhead together exceed what one GPU's HBM holds, not because more GPUs are inherently faster. Work out that number first, at your real context length and concurrency target, before you look at a spec sheet.

Weights are the easy part of the math. FP16/BF16 costs 2 bytes per parameter, so a 70B model needs roughly 140GB, already past the 80GB ceiling on a single H100 SXM5 or A100 SXM4. INT8 halves that to 70GB, still tight on an 80GB card once you add cache and overhead. INT4 gets you to about 46GB, which fits on a single 80GB GPU with room to spare.

KV cache is the part teams underestimate. It grows linearly with context length and concurrent requests, not with model size alone. For Llama 3.1 70B at BF16, a single 128K-token request adds roughly 40GB of KV cache on top of the 140GB of weights, pushing the total footprint for that request to about 180GB, with cache alone accounting for roughly 22% of it. Serve four such requests concurrently and the cache alone can exceed the weights.

The counter-argument worth taking seriously: adding GPUs doesn't fix decode latency if your workload is memory-bandwidth-bound rather than VRAM-capacity-bound. During single-request decode, the GPU spends most of its time streaming weights from HBM rather than computing, so splitting those same weights across more GPUs doesn't remove the bandwidth ceiling, it just moves it and adds synchronization on top. Our guide to the AI memory wall problem covers why "add more GPUs" is the wrong first move for a latency problem that's actually about batching and bandwidth. Use the VRAM math above to decide if you're forced onto multiple GPUs at all; treat throughput scaling as a separate, secondary question.

Tensor Parallelism vs Pipeline Parallelism, Explained for Non-Infra Readers

Once VRAM forces you past one GPU, the next decision is how to split the model, and that choice is dictated by what network link connects your GPUs. Tensor parallelism (TP) splits the math inside each layer across GPUs: every GPU computes a different slice of the same layer on the same input, then the results need to be combined with an all-reduce before the next layer can run. Llama 70B has 80 transformer layers, and each one typically needs two all-reduce calls, so a single forward pass triggers roughly 160 separate synchronization points. TP needs frequent, high-bandwidth communication and is almost always kept inside a single NVLink-connected node.

Pipeline parallelism (PP) splits whole layers across GPUs instead of splitting the math within a layer. Data flows through the stages like an assembly line, and each stage only passes activations to the next one, a point-to-point transfer rather than an all-to-all synchronization. That lighter communication pattern is why pipeline parallelism tolerates slower interconnects, including InfiniBand between nodes or even PCIe, where tensor parallelism falls over.

We keep coming back to NVIDIA's Megatron-LM paper, which established the parallelism playbook most training frameworks still follow: it showed how tensor, pipeline, and data parallelism compose to scale to thousands of GPUs, with the now-standard pattern of keeping tensor parallelism inside a node and using pipeline and data parallelism to stretch across nodes. Pipeline parallelism does introduce its own tax, the "bubble" where later stages sit idle waiting for the pipeline to fill and drain, but the GPipe analysis cited in Lilian Weng's survey on training large models found the bubble becomes close to negligible once the number of microbatches exceeds roughly 4x the number of pipeline stages, which is a batching decision, not a hardware one.

For a hands-on setup with FSDP, DeepSpeed ZeRO-3, and Megatron-Core, our distributed LLM training guide covers the multi-node configuration this section only summarizes.

Cost Comparison: One H200 vs a Multi-GPU A100 Cluster for the Same Model

The cheaper cluster on paper isn't always the cheaper cluster in practice, because a wider tensor-parallel domain adds communication overhead the hourly rate doesn't show. We see teams anchor on the per-GPU sticker price and skip this step, then get surprised when the "cheaper" cluster loses on tokens-per-dollar once it's actually serving traffic. Here's what that trade-off actually looks like priced out.

Live on-demand pricing on Spheron as of 14 Aug 2026:

Pricing fluctuates based on GPU availability. The prices above are based on 14 Aug 2026 and may have changed. Check current GPU pricing → for live rates.

Take Llama 3.1 70B at BF16, ~140GB of weights, deployed for interactive inference with moderate context. Two paths:

• 2x H200 SXM5 (282GB pool, NVLink 4 at 900GB/s per GPU): weights consume half the pool, leaving well over 100GB spread across two GPUs for KV cache and overhead. At the current range, that's roughly $9.68-$12.18/hr for the pair, and tensor parallelism only has to synchronize across two GPUs per layer.
• 4x A100 80GB (320GB pool, NVLink 3 at 600GB/s total): at the low end of the current range, around $1.43/hr each, that's about $5.72/hr, noticeably cheaper on the invoice. But it needs 4-way tensor parallelism instead of 2-way, which means more all-reduce rounds per layer, over an interconnect with about a third less aggregate bandwidth than the H200 pair.

For latency-sensitive interactive serving, the extra synchronization on the wider, slower-linked A100 cluster can eat into the hourly savings once you measure tokens-per-dollar rather than dollars-per-hour. For throughput-bound batch inference, where communication overhead amortizes across bigger batches, the A100 cluster's lower sticker price usually wins outright. Benchmark your actual workload before committing either way; the A100 vs H100 architecture and pricing breakdown covers the compute and bandwidth gap between the two GPU generations in more depth than fits here.

If your VRAM math from the first section puts you at or under 141GB, a pair of H200 GPU rental instances is the simpler build: half the synchronization domain of a 4-GPU alternative, at a real dollar premium that's smaller than it looks once you account for wall-clock time. If your model comfortably fits INT8 or the cheaper hourly rate matters more than raw latency, on-demand A100 pricing on Spheron is worth pricing out directly against the H200 pair for your specific batch size.

NVLink vs InfiniBand: What Networking Overhead Actually Costs a GPU Cluster for AI

NVLink and InfiniBand aren't competing choices, they solve different halves of the same cluster: NVLink connects GPUs inside a node, InfiniBand connects nodes to each other, and the bandwidth gap between them is roughly an order of magnitude. NVLink 4 on H100 and H200 SXM delivers up to 900GB/s aggregate bandwidth per GPU. NDR InfiniBand, the fabric most multi-node clusters use for inter-node links, tops out around 400Gb/s, about 50GB/s, per link. That gap is exactly why tensor parallelism stays inside a node and pipeline parallelism crosses nodes.

How far can a single NVLink domain stretch before you need InfiniBand at all? NVIDIA's GB200 NVL72 connects 72 Blackwell GPUs into one NVLink domain with up to 130 TB/s of aggregate NVLink bandwidth, which is most of a modern supercomputer's GPU count without ever leaving NVLink. Most teams renting GPU cloud capacity won't reach that scale, but it's a useful anchor for how much scaling headroom NVLink alone now covers before InfiniBand becomes mandatory.

The network choice shows up directly in training step time. For a 70B model, an all-reduce moving 245GB of gradients across 8 data-parallel ranks takes about 4.9 seconds at 400Gbps InfiniBand versus 19.6 seconds at 100GbE, a 4x difference from the network alone, with no change to the model or the GPUs. Our NCCL tuning guide for multi-GPU training walks through the environment variables that get you close to that theoretical InfiniBand number in practice.

InfiniBand also costs real money to build, and it's worth pricing before you assume it's mandatory. According to Introl's InfiniBand vs. Ethernet cost breakdown, a 1,000-GPU cluster runs roughly $15M in InfiniBand hardware versus about $7M for equivalent Ethernet, an $8M premium driven largely by per-port adapter costs: $2,000-3,000 for InfiniBand NICs versus $800-1,500 for comparable Ethernet cards. If your parallelism strategy is pipeline- and data-parallel across nodes rather than tensor-parallel, the communication pattern is lighter and a cheaper fabric may hold up fine; our guide to multi-node GPU training without InfiniBand covers exactly that trade-off and when it's a reasonable call versus a mistake. For a deeper comparison of InfiniBand, RoCE, and Spectrum-X, see our GPU networking decision guide.

NVIDIA CEO Jensen Huang put the underlying problem plainly in a conversation with Lex Fridman: "So you added 10,000 computers, but you would like it to go a million times faster. Then all of a sudden you have to take the algorithm, you have to break up the algorithm, you have to refactor it, you have to shard the pipeline, you have to shard the data, you have to shard the model." More GPUs alone don't buy proportional speed. Matching your parallelism strategy to the fabric that actually connects your GPUs is what turns extra hardware into extra throughput instead of extra synchronization overhead.

Sizing a GPU Cluster for AI: A Checklist by Model Size and Traffic

Model size sets the floor, but traffic (context length and concurrency) is what usually pushes a deployment from "fits on one GPU" to "needs a cluster" in production, even when the base model would fit at low traffic.

Two traffic variables move a model between rows in that table faster than parameter count does. Context length adds KV cache linearly, so a 32B model at 4K context and the same model at 128K context can differ by tens of gigabytes of cache alone. Concurrency multiplies that cache by the number of simultaneous requests you actually serve, which is why a model that "fits" in a load test at concurrency 1 can force a cluster at production concurrency 20. Size for your real traffic pattern, not the model card's minimum requirement.

If you haven't settled on a GPU model yet, our best NVIDIA GPUs for LLMs by use case ranks the current lineup, and H100 GPU specs and pricing is worth checking directly if your sizing lands you at the 80GB tier rather than H200's 141GB.

Quick Decision Framework: Single Large GPU or Multi-GPU Cluster

1. Add up the real number. Weights (at your target precision) plus KV cache (at your real context length and concurrency) plus 15-20% framework overhead. If that fits inside one GPU's VRAM with 20-30% headroom, stop, you don't need a cluster.
2. If it doesn't fit, stay inside one node first. Use tensor parallelism across the NVLink-connected GPUs in a single bare-metal node before you ever cross a network boundary. This is the highest-bandwidth, lowest-latency option you have.
3. If you must cross nodes, switch strategy, not just hardware. Don't extend tensor parallelism over InfiniBand, it needs bandwidth InfiniBand doesn't have. Move to pipeline or data parallelism at the node boundary, and batch enough microbatches (above roughly 4x your pipeline stage count) to keep the pipeline bubble from eating your gains.
4. Price the cluster on real per-GPU rates and real communication overhead, not sticker VRAM. A cheaper GPU with a wider tensor-parallel domain can cost more in wall-clock time than a smaller, pricier NVLink-dense pair, depending on whether your workload is latency- or throughput-bound.

Once you know your shape, whether that's a single H200, a pair of them, or a wider A100 cluster, tuning the interconnect is what actually delivers the throughput the extra hardware promised.

If your VRAM math points to a cluster instead of one GPU, get the interconnect right before you scale further: bare-metal NVLink access and per-minute billing on Spheron make it cheap to benchmark tensor parallelism against pipeline parallelism on your real model before committing to a build-out.

Rent H200 → | Spheron A100 instances → | View all pricing →