Tensor parallelism vs pipeline parallelism comes down to which axis you cut a model along: split the math inside a layer, or split the layers themselves. Get that choice wrong on a rented GPU cluster and you pay for GPUs that spend more time synchronizing than computing. We'll work through both mechanisms from first principles, then resolve the whole comparison to the one decision that actually matters when you're paying by the hour: which interconnect tier, PCIe, NVLink, or InfiniBand, your parallelism plan requires you to rent.
For the VRAM math that decides whether you're forced onto multiple GPUs at all, our GPU cluster sizing guide covers that decision in depth. This post picks up from there: once you know you need more than one GPU, here's how to split the model and what fabric that split actually requires.
Why One GPU Isn't Enough for Large Models
A model's weights have to live somewhere, and "somewhere" is a fixed pool of GPU memory: 80GB on an H100, 141GB on an H200. Run the arithmetic on a 70B-parameter model at 16-bit precision (2 bytes per parameter) and weights alone need roughly 140GB, before a single token of KV cache lands in memory. That already exceeds every single-GPU SKU except the H200, and even there it leaves no room for a long-context request's KV cache or the serving framework's own overhead.
The full weight-plus-cache derivation, tier by tier across quantization levels, lives in our LLM VRAM requirements guide, so this post won't re-derive it. The short version: once total memory demand exceeds what one GPU holds, there are exactly two ways to spread a model across more than one, and they answer two different questions. Tensor parallelism asks "how do I split the math inside a layer so it fits?" Pipeline parallelism asks "how do I split the layers themselves across GPUs?" Both get you past the VRAM ceiling. They get there through completely different communication patterns, and that difference is what decides which network fabric you need underneath.
Tensor Parallelism: Splitting Layers Wide
Tensor parallelism (TP) cuts a layer's own computation into pieces and hands one piece to each GPU. Every GPU in the group processes the same input tokens simultaneously, each computing on its own shard of the weight matrix, and the partial results then have to be combined before the next layer can start.
How TP Shards a Weight Matrix and Why Every Layer Needs an All-Reduce
A transformer layer's linear projections, the attention QKV matrices and the MLP's up and down projections, are large weight matrices multiplied against the input activations.
That synchronization is an all-reduce: every GPU in the tensor-parallel group sends its partial result to every other GPU and sums them, so each GPU ends the layer holding the same complete output before the next layer runs. Because this happens after (typically) both the attention block and the MLP block in every transformer layer, tensor parallelism turns a forward pass into a long chain of frequent, small, latency-sensitive synchronizations rather than one big transfer at the end. That's the trade TP makes: it needs less memory per GPU, but it needs a fast, low-latency link between every GPU doing the work, which is exactly why it's normally kept inside a single node.
Worked Example: TP=2 vs TP=8 on a 70B-Class Model
What changes is how many GPUs each all-reduce has to touch, and how much memory each GPU has to hold.
At TP=2, the 140GB weight set splits into two ~70GB shards, so you need two 80GB GPUs at minimum with almost no headroom left over. Each all-reduce is a simple two-way exchange. It's the smallest possible TP group, cheap to synchronize, but it barely helps with memory: you've traded a single GPU that doesn't fit the model for two GPUs that barely do.
At TP=8, the same 140GB splits into eight ~17.5GB shards, leaving most of each GPU's memory free for KV cache, longer context, or higher concurrency. But now every all-reduce is an eight-way collective spanning every GPU in the group, and ring-based all-reduce algorithms get more communication rounds, not fewer, as the participant count grows. That's the reason TP=8 is the practical ceiling for tensor parallelism on current hardware: eight GPUs is exactly the size of a standard NVLink-connected SXM5 node, and going wider means the all-reduce has to leave the node.
Pipeline Parallelism: Splitting Layers Deep
Pipeline parallelism (PP) takes the opposite cut. Instead of splitting what happens inside a layer, it splits which GPU owns which layers. GPU 0 might hold layers 1 through 20, GPU 1 holds layers 21 through 40, and so on, so each GPU runs a full, unmodified slice of the model rather than a fraction of every layer.
How Stages Pass Activations Instead of Synchronizing Weights
Data flows through pipeline stages the way parts move down an assembly line. GPU 0 computes its 20 layers on a micro-batch of input, then sends only the resulting activations to GPU 1, which picks up where GPU 0 left off.
This is a fundamentally lighter communication pattern than TP's all-reduce. Pipeline parallelism's point-to-point activation handoff moves far less data per exchange than an all-reduce, and it only ever talks to the adjacent stage, which is exactly why it tolerates a slower interconnect between nodes where tensor parallelism would stall.
The Pipeline Bubble: Where the Idle Time Comes From, and How Microbatch Count Fixes It
Pipeline parallelism's tax isn't bandwidth, it's idle time. At the start of a batch, only GPU 0 has work while every downstream stage waits for the first micro-batch to arrive. At the end, GPU 0 has finished while downstream stages are still draining. That idle window is the "pipeline bubble."
The fill-and-drain window is roughly fixed no matter how big the batch is, so the lever you have is the number of microbatches you push through it. More microbatches means more useful work happens between the fill and the drain, so that fixed idle window shrinks as a share of total runtime. It's a batch-size and scheduling decision you make in software, not a hardware upgrade you have to buy.
Tensor Parallelism vs Pipeline Parallelism at a Glance
| Dimension | Tensor Parallelism | Pipeline Parallelism |
|---|---|---|
| What it splits | The math inside each layer (weight matrix shards) | Whole layers, grouped into stages |
| Communication pattern | All-reduce across the full TP group | Point-to-point activations to the next stage |
| Communication frequency | Roughly every layer (attention + MLP) | Once per stage boundary |
| Main overhead | All-reduce latency and bandwidth | Pipeline bubble (idle GPU time) |
| Bandwidth requirement | High, needs NVLink-class links | Modest, tolerates InfiniBand or PCIe |
| Typical scope | Inside one NVLink-connected node | Across nodes, or across a whole rack |
| How you fix its overhead | Fewer participants per all-reduce, faster fabric | More microbatches per pipeline stage |
| Framework default (multi-node) | Sized to GPUs per node | Sized to number of nodes |
Interconnect Requirements and What They Cost
The parallelism strategy you pick isn't really a modeling decision, it's a networking decision, because TP and PP have such different tolerance for link speed. TP needs NVLink-class intra-node bandwidth to keep its all-reduces from becoming the bottleneck; PP's lighter activation transfers run fine over InfiniBand between nodes, and often over cheaper fabric still.
NVLink vs PCIe vs InfiniBand: Which Parallelism Strategy Needs Which Fabric
The bandwidth gap between these three tiers is the entire reason the TP/PP split exists. Cross a node boundary and the fabric changes: Spheron's own multi-node H200 spec lists 400 Gb/s NDR InfiniBand across nodes, a figure NVIDIA's Quantum-2 InfiniBand switch line confirms as its top port speed, which works out to roughly 50 GB/s once you convert from bits to bytes, well below NVLink's 900 GB/s but still enough for PP's lighter traffic.
That ordering maps directly onto the two parallelism strategies: TP's frequent all-reduces need NVLink-class bandwidth to avoid becoming the bottleneck, while PP's occasional point-to-point activation handoff tolerates InfiniBand's lower ceiling, and in workloads with light cross-node traffic, even a cheaper Ethernet fabric can hold up. A recent measurement study on distributed LLM serving found exactly this asymmetry in practice: "tensor parallelism incurs substantial network overhead but delivers superior response times for brief sequences, pipeline parallelism minimizes data transfer requirements while increasing total latency, and combined approaches demand careful tuning to achieve balanced performance."
What Renting Each Interconnect Tier Actually Costs
On its H100 SXM5 availability page, Spheron confirms the H100 comes in both SXM5 and PCIe form factors side by side, so the interconnect tier you get depends on which SKU you provision, not a separate published price line. As of 02 Sep 2026, Spheron lists H100 SXM5 (80GB) at $5.07/hr on-demand and $2.91/hr spot, and H200 SXM5 (141GB) at $5.92/hr on-demand and $3.31/hr spot, with SXM5 the form factor that carries NVLink and PCIe the one that doesn't. If your parallelism plan needs NVLink for TP, confirm you're provisioning the SXM5 SKU rather than assuming it from the GPU name alone. Custom multi-node clusters scale from 8 to 512+ GPUs, with InfiniBand configs available on request.
That's the honest trade-off worth naming: the single-GPU and single-node tiers are self-serve in minutes, but a multi-node InfiniBand build isn't an instant click-to-deploy product the way a single H100 or H200 is, and published pricing for large custom clusters isn't posted the way per-GPU on-demand rates are.
Pricing fluctuates based on GPU availability. The prices above are based on 02 Sep 2026 and may have changed. Check current GPU pricing → for live rates.
Combining Both: Why Production Clusters Run TP Inside a Node and PP Across Nodes
Real deployments rarely pick just one. The standard pattern, set by Megatron-LM and now the default recipe in most serving frameworks, is tensor parallelism inside a node and pipeline (or data) parallelism to stretch across nodes. TP stays where the fast fabric is; PP crosses the boundary where the fabric gets slower.
TP=8 stays inside each NVLink-connected node; PP=2 is the only thing that crosses the InfiniBand link between them.
At real scale, this composability is what makes trillion-parameter training tractable at all. The Megatron-LM paper reports training a 1-trillion-parameter model at 502 petaFLOP/s across 3,072 GPUs, reaching 52% of theoretical peak per-GPU throughput, by combining tensor, pipeline, and data parallelism rather than relying on any single one to scale that far. The same paper's interleaved pipeline schedule, which changes the order stages process microbatches rather than adding new hardware, improves throughput by more than 10% at a comparable memory footprint over the standard schedule, purely from a better scheduling algorithm. Combining parallelism strategies gets you the scale; tuning how they're scheduled is what gets you the throughput once you're there.
Tensor Parallelism vs Pipeline Parallelism: Choosing a Strategy for a Rented GPU Cluster
A Decision Path: Model Size, Latency vs Throughput, and GPU Count
- Check if you need to split at all. If weights, KV cache, and framework overhead fit on one GPU with 20 to 30% headroom, you don't need TP or PP. Stop here, that's the mistake we see most often: teams reach for multi-GPU parallelism before confirming a single card can't do the job.
- If you're splitting within a single node, default to tensor parallelism. Size the TP group to the GPUs you have in that node (commonly 2, 4, or 8) and make sure it's an NVLink SKU, not PCIe.
- If your model needs more GPUs than fit in one node, add pipeline parallelism at the node boundary. Don't extend tensor parallelism across nodes; InfiniBand doesn't have the bandwidth TP's all-reduce needs.
- Run enough microbatches to bury the pipeline bubble. The fill-and-drain window at the start and end of a batch is roughly fixed regardless of batch size, so pushing more microbatches through the pipeline before it drains shrinks that fixed cost as a share of total runtime. Start from your framework's default and increase it until throughput stops improving.
- Weigh latency against throughput. TP-heavy setups tend to win on response time for shorter requests because there's no pipeline fill-and-drain delay; PP-heavy setups tend to win on aggregate throughput once batches are large enough to keep every stage busy, matching the pattern the arXiv measurement study above found in production serving traffic.
Multi-GPU Inference Frameworks and Which Parallelism They Default To
- Megatron-LM. The framework this whole playbook traces back to. It composes tensor, pipeline, and data parallelism explicitly, and its interleaved pipeline schedule is what gets that extra 10%+ throughput at trillion-parameter scale. For a hands-on setup with Megatron-Core alongside FSDP and DeepSpeed ZeRO-3, our distributed LLM training guide covers the multi-node configuration this post only summarizes.
If you're serving a mixture-of-experts model, TP and PP aren't the whole picture: expert parallelism is a third way to split a model that routes tokens to specific GPUs rather than splitting math or layers, and it's worth knowing it exists before you force an MoE model into a TP/PP-only plan. Our MoE inference optimization guide covers it in depth. For long-context workloads, sequence parallelism is a fourth axis again, splitting the sequence dimension itself rather than the model, covered in our ring attention and tree attention guide. For a worked example that makes an explicit TP=8-vs-TP=4-plus-PP=2 call on a real 550B model, see our Nemotron 3 Ultra deployment guide. And once you've picked a serving stack to actually run TP or PP on, our vLLM vs TensorRT-LLM vs SGLang benchmarks compares them head to head on the same hardware.
Whichever split you land on, tune the collective communication underneath it before you conclude the strategy itself is slow: our NCCL tuning guide for multi-GPU training walks through the environment variables that get real hardware closer to the theoretical bandwidth numbers cited throughout this post. Full setup instructions for provisioning the underlying instances are in the Spheron docs.
Whichever split your model needs, the interconnect underneath it is what turns the plan into real throughput: Spheron rents SXM5 H100 and H200 instances with NVLink between GPUs for tensor parallelism, plus custom multi-node clusters with InfiniBand for pipeline parallelism across nodes.
Spheron H100 → | Spheron H200 clusters → | View all pricing →
Frequently Asked Questions
Tensor parallelism splits the math inside each layer across GPUs, so every GPU computes a slice of the same layer and the results need an all-reduce before the next layer can run. Pipeline parallelism splits whole layers across GPUs instead, so each GPU owns a contiguous block of layers and only passes activations to the next stage. TP's all-reduce pattern needs NVLink-class bandwidth; PP's point-to-point activation transfers tolerate InfiniBand or even PCIe.
In practice, yes, for anything beyond two GPUs. Every layer's all-reduce moves the full activation tensor between every GPU in the tensor-parallel group. NVLink 4.0 moves that at up to 900 GB/s per GPU; PCIe Gen5 tops out at 128 GB/s, roughly 7x slower, so the all-reduce turns into the bottleneck and starts eating the throughput gain you added GPUs to get.
A pipeline bubble is the idle time GPUs spend at the start and end of a batch while the pipeline fills and drains, since a later stage can't start until an earlier one hands it activations. The fix is scheduling, not hardware: the fill-and-drain time stays roughly fixed regardless of batch size, so running more microbatches through the pipeline before it drains means that fixed idle window shrinks as a share of total runtime.
vLLM's documented multi-node recipe sets tensor_parallel_size to the number of GPUs per node and pipeline_parallel_size to the number of nodes, which keeps TP's all-reduce traffic inside a single node's fast interconnect and lets PP absorb the slower link between nodes.
Technically yes, but production stacks avoid it by design. vLLM's own docs recommend switching to pipeline parallelism instead of tensor parallelism specifically when a node's GPUs lack NVLink, such as L40S nodes, because pipeline parallelism gets higher throughput and lower communication overhead in that setup. The same logic holds once you cross a full node boundary: tensor parallelism's frequent all-reduce traffic wants the fastest fabric available, and pipeline parallelism is what's built to tolerate the slower link instead.
