Ring Attention and Tree Attention on GPU Cloud: Sequence Parallelism for 10M-Token LLM Inference (2026 Guide)

Past 1M tokens, the standard tricks stop working. NVMe KV offloading tops out around 500K tokens before NVMe bandwidth itself becomes the bottleneck. Prefill-decode disaggregation helps throughput but doesn't reduce per-request KV memory. Serving Qwen 3.6 Plus or Kimi K2.6 at their advertised 4M-10M token contexts requires distributing the attention computation itself across GPUs. That's sequence parallelism.

For the foundations, see our KV Cache Optimization guide and NVMe KV Cache Offloading guide.

The 1M-Token Memory Wall

KV cache memory scales linearly with context length. At 128K tokens, a single Llama 3.1 70B request at BF16 uses ~43 GB just for KV. At 1M tokens, that same request needs ~328 GB. No single GPU has that much HBM, and NVMe bandwidth can't keep up once you scale to multi-million-token contexts.

Here's the math for a 70B-class model (80 layers, 8 GQA heads, 128 head dim):

The formula: KV_bytes = 2 × L × H_kv × D × S × bytes_per_element

At 1M tokens with FP8 quantization (~164 GB), you need at least 2-3 H200s (141 GB each) just for the KV cache of a single request, leaving almost no room for model weights. At 4M tokens, FP8 KV alone exceeds 4-5 H200s combined HBM.

The bottleneck shifts depending on scale:

• Under 500K tokens: HBM is the binding constraint. NVMe offloading at 7 GB/s works because cold blocks load faster than recomputing them. This is exactly what NVMe KV Cache Offloading covers.
• 500K-4M tokens: NVMe bandwidth becomes the bottleneck. Even with fast NVMe (7 GB/s), loading ~164-655 GB of FP8 KV data per request takes seconds per token generation step.
• 4M-10M tokens: There is no single-node solution. The KV cache physically cannot fit in a reasonable number of GPUs' combined HBM unless you distribute the attention computation itself.

Sequence parallelism is the solution at the upper end of this curve. Instead of storing the full KV cache on every GPU, you shard it across a ring of GPUs so each one holds 1/N of the sequence.

Ring Attention vs Tree Attention vs Striped Attention

Three variants of sequence parallelism exist, each with different communication topology and performance tradeoffs:

Ring Attention rotates KV shards around the ring one step per round, computing local attention while data is in transit. The communication-compute overlap is excellent when chunk sizes are tuned right. But for decoder-only (causal) models, the triangular attention mask means GPU 0 (holding position 0-128K) computes dense attention while GPU 7 (holding position 896K-1M) computes only ~12% as many non-masked blocks. That load imbalance wastes 3-4x GPU time on late-position ranks.

Tree Attention uses a binary reduce tree: each GPU computes local attention, then adjacent pairs merge attention outputs using log-sum-exp stabilization. The number of communication rounds drops from N to log2(N), which matters at ring sizes above 32. The tradeoff is that each merge step requires numerically stable softmax combination, adding implementation complexity and some accuracy risk at low precision.

Striped Attention (Brandon et al., 2023) fixes Ring Attention's load imbalance by assigning non-contiguous sequence chunks to each GPU. Instead of GPU 0 holding tokens 0-128K and GPU 7 holding 896K-1M, each GPU holds interleaved positions (GPU 0: 0, 8, 16, 24...; GPU 1: 1, 9, 17, 25...; etc.). Under the causal mask, every GPU now handles an approximately equal mix of dense early-position blocks and sparse late-position blocks. GPU utilization is balanced across the ring.

The recommendation: Use Striped Attention for all causal decoder-only models. Plain Ring Attention is fine for bidirectional attention (encoders, Gemma 3's sliding window for prefill). Tree Attention is worth exploring only when ring size exceeds 32 and latency rather than throughput is the primary constraint.

Hardware Requirements

Sequence parallelism is interconnect-limited. The ring communication volume per ring step scales with the KV shard size: moving a 1/N chunk of the sequence's KV tensors across the ring. The bandwidth you have determines the maximum ring size and context length you can serve at acceptable GPU utilization.

For single-node ring attention on bare-metal H200 instances, NVLink 4.0 at 900 GB/s is more than adequate. An 8-GPU ring can handle 1M-token contexts for 70B-class models with FP8 KV, with per-ring-step transfer times well under 2ms for chunk sizes up to 8K tokens.

For multi-node rings targeting 4M-10M token contexts, InfiniBand NDR is the practical requirement. RoCEv2 is technically viable at ring sizes of 8-16, but latency climbs fast above that. If you're building a 32+ GPU ring for 10M-token inference, InfiniBand NDR or NVLink 5.0 Switch is not optional.

The chunk size calculation: for any interconnect, the per-step transfer volume (KV shard size for one chunk) divided by the available bandwidth should stay under 10ms to maintain above 80% GPU compute utilization. For InfiniBand NDR at ~350 GB/s:

Max transfer per step = 10ms × 350 GB/s = 3.5 GB
Max chunk at FP8 (70B, 0.163 MB/token) = 3.5 GB / 0.163 MB = ~21,500 tokens

This means 16K-21K tokens per chunk is the practical sweet spot for multi-node InfiniBand rings with FP8. Going to 32K-token chunks at 350 GB/s gives ~15ms per step, still within acceptable range for FP8 workloads.

For the full interconnect cost analysis, see our GPU Networking guide: InfiniBand vs RoCE vs Spectrum-X. For Spheron cluster instance configurations with InfiniBand NDR fabric, see the Spheron cluster instance types docs.

Setting Up Ring Attention

Megatron-Core with transformer_engine

Megatron-Core's context parallelism (CP) implementation supports Ring Attention with configurable chunking strategies including striped layout. Install from source:

pip install git+https://github.com/NVIDIA/Megatron-LM
pip install transformer_engine[pytorch]

Core config for Ring Attention with striped layout on a Qwen 3.6 Plus-scale model:

model_config = {
    "context_parallel_size": 8,        # ring size, must divide total GPUs
    "sequence_parallel": True,          # activates SP communication hooks
    "use_flash_attn": True,             # FA4 on Blackwell, FA3 on Hopper
    "cp_comm_type": "p2p",              # peer-to-peer ring communication
}

For multi-node setups, set these NCCL environment variables before launching your job. See our NCCL Tuning guide for the full context:

export NCCL_ALGO=Ring
export NCCL_NTHREADS=512
export NCCL_IB_HCA=mlx5_0,mlx5_1    # adjust to your IB HCAs
export NCCL_IB_GID_INDEX=3            # RoCEv2 GID index if applicable

Place context-parallel ranks on the same physical node where possible. Two CP ranks on the same node communicate over NVLink at 900 GB/s; cross-node CP ranks use InfiniBand at ~350 GB/s. The performance difference is about 2.5x. For a 16-GPU CP ring across two nodes, NVLink handles intra-node pairs, InfiniBand handles cross-node pairs. Design your parallelism topology to minimize cross-node CP communication.

transformer_engine handles kernel selection automatically: FA4 on SM100 (B200, B300), FA3 on SM90 (H100, H200), FA2 on older architectures. No manual kernel selection required within each CP rank's local compute step.

For more on Megatron-Core setup and parallelism configs, see Distributed LLM Training: FSDP, DeepSpeed, and Megatron-Core multi-node.

SGLang Sequence-Parallel Serving

SGLang v0.4+ implements CP with an all-gather + local attention pattern, equivalent to Ring Attention for inference. The interface is a single launch flag:

python -m sglang.launch_server \
  --model-path Qwen/Qwen3.6-Plus \
  --tp 8 \
  --context-parallel-size 4 \
  --context-length 1000000 \
  --dtype bfloat16 \
  --mem-fraction-static 0.85 \
  --chunked-prefill-size 8192

This example: 32 total GPUs (TP=8, CP=4), 4 nodes, 1M-token context. The --context-parallel-size flag activates the ring-attention communication schedule with striped layout for causal models. --chunked-prefill-size 8192 interleaves 8K-token prefill chunks with decode steps, preventing long prefills from blocking ongoing decode requests.

For a B200 multi-GPU cluster, add --enable-fp8-kv to halve KV memory per GPU. Combined with CP=4, a single 1M-token request distributes its ~164 GB FP8 KV cache across 4 CP ranks (32 GPUs total), each CP rank holding ~41 GB, well within B200's 192 GB HBM per GPU.

For B200 GPU rental on a multi-node cluster, check B200 GPU rental for availability and cluster configuration options.

FlashAttention-4 Integration and Chunked Prefill

FlashAttention-4 activates automatically within each CP rank on Blackwell GPUs (B200, B300) via transformer_engine and vLLM's FlashAttention dispatch. No manual kernel selection needed. Each GPU's local attention chunk (1/N of the full sequence) runs the SM100 tile-based FA4 kernel, which reduces HBM read overhead by roughly 2x compared to FA3. The FA4 gains compound with CP: you get FA4's per-token efficiency on each chunk, plus CP's memory distribution across the ring.

For the full FA4 architecture explanation and migration guide, see FlashAttention-4 on GPU Cloud: Blackwell Inference Guide.

Chunked prefill with sequence parallelism processes the input prompt in fixed-size chunks. Each chunk is processed ring-wise across all CP ranks before moving to the next chunk. This matters for two reasons:

1. TTFT for blended workloads: Without chunked prefill, a single long-context prefill can block the decode queue for seconds. Chunked prefill lets the decode queue continue while prefill progresses chunk by chunk.
2. Memory footprint: Processing 1M-token prompts in 8K-token chunks limits peak activation memory on each rank. Without chunking, the intermediate activation buffer scales with the full chunk size seen by each rank (1M / N tokens), which can OOM on 80 GB GPUs.

Recommended chunk sizes:

• Bare-metal B200 instances + NVLink 5.0 (intra-node): 4K-8K tokens per chunk
• B200 + InfiniBand NDR (multi-node): 8K-16K tokens per chunk
• On-demand H200 access + NVLink 4.0 (intra-node): 4K-8K tokens per chunk

For blended workloads (concurrent prefill + decode), set --chunked-prefill-size 8192 in SGLang or --enable-chunked-prefill --max-num-batched-tokens 8192 in vLLM.

Tuning: Ring Size, Chunk Size, and Overlap

Three knobs control performance of a ring-attention setup:

Ring Size (CP Degree)

Ring size determines how the sequence is sharded. More GPUs per ring means smaller KV shard per GPU, but more communication rounds and more inter-GPU coordination.

The key constraint: TP degree is bounded by the number of KV heads. If your model has 8 GQA heads, max TP = 8. CP has no such bound. For very large context lengths, pushing CP higher while keeping TP at max is usually the right direction.

Chunk Size and the 10ms Rule

Chunk size is the number of tokens in each ring communication step. The rule: chunk transfer time must stay under 10ms to keep GPU utilization above 80%.

chunk_transfer_time = (chunk_tokens × KV_bytes_per_token) / interconnect_bandwidth_GBs

For B200 (FP8, 70B: 0.163 MB/token) + InfiniBand NDR (350 GB/s):

For multi-node InfiniBand rings with FP8, 16K-21K tokens per chunk is the practical sweet spot. At 4K tokens you get excellent overlap but more kernel launch overhead; at 32K tokens transfer time climbs to ~15ms, starting to cut into compute utilization.

For intra-node NVLink at 900-1800 GB/s, even 32K-token chunks have under 8ms transfer time, so intra-node rings are not chunk-size-sensitive in the same way.

Parallelism Decomposition

The full parallelism equation: total GPUs = TP × CP × PP (tensor parallel × context parallel × pipeline parallel). For inference, PP is usually 1 (pipeline stages add latency for single requests). So:

At 4M tokens, 32x B200 (InfiniBand NDR):

• Option A: TP=8, CP=4. Four 8-GPU TP groups, each with 4-way CP. Easy to configure in SGLang and Megatron.
• Option B: TP=4, CP=8. Smaller TP means more communication for weight sharding, but larger CP accommodates longer sequences per ring step.
• Benchmark both. The crossover depends on the model's attention head count and the specific batch workload.

Benchmarks and Cost Analysis

These figures are estimated performance ranges based on hardware interconnect specs and published FlashAttention and SGLang CP benchmarks. They are not measured results from a specific Spheron deployment. Run your own benchmarks on your target hardware for production planning.

B200 pricing: $7.00/GPU/hr on-demand ($1.71/GPU/hr spot). H200 pricing: $4.72/GPU/hr on-demand.

Cost per million output tokens (on-demand, midpoint throughput estimates):

Long-context inference at 10M tokens is expensive. That's not a Spheron pricing issue; it reflects the physics of attention: every decode step reads O(N) KV tokens from HBM. At 10M tokens, a single forward pass reads ~1.6 TB of FP8 KV data (distributed across 32 Spheron B200 GPUs, ~50 GB per GPU per step). The compute is real. Sequence parallelism makes it possible; it doesn't make it cheap.

For live rates, check current GPU pricing.

Pricing fluctuates based on GPU availability. The prices above are based on 14 May 2026 and may have changed. Check current GPU pricing → for live rates.

Decision Matrix: When to Use Each Approach

The SRAM chip row covers purpose-built inference accelerators with large on-chip SRAM that avoids HBM entirely. NVIDIA's Rubin CPX was announced in this category before being replaced. For the current state of that hardware category, see NVIDIA Rubin CPX and long-context inference.

The crossover point between NVMe KV offloading and sequence parallelism is around 500K-1M tokens. Below 500K, NVMe offloading covers the gap at much lower cost. Above 1M, sequence parallelism is the only practical option.

Sequence parallelism also complements prefill-decode disaggregation at extreme context lengths. Disaggregation addresses the prefill/decode resource mismatch; sequence parallelism addresses the per-request memory constraint. At 4M+ token workloads you likely need both.

For H200 SXM5 bare-metal instances for single-node ring setups, capacity is available at $4.72/hr per GPU. See H200 SXM5 instances on Spheron for the current configuration options.

Practical Checklist Before Running Ring Attention in Production

1. Verify interconnect type before provisioning. Multi-node ring attention requires InfiniBand NDR or NVLink Switch. Standard Ethernet (even at 400GbE RoCEv2) adds enough latency at ring sizes above 16 to drop GPU utilization below 70%.

2. Calculate KV-per-GPU before you launch. The formula: (2 × L × H_kv × D × S × bytes) / N. Plug in your model's architecture constants. If the result exceeds 80% of HBM capacity, add more nodes or reduce context length.

3. Use Striped Attention for decoder-only models. In Megatron-Core: set context_parallel_size=N and cp_comm_type=p2p; striped sequence layout is the default for causal models in recent Megatron-Core releases. In SGLang: --context-parallel-size N uses striped layout for causal models by default.

4. Tune chunk size to the 10ms rule. For InfiniBand NDR, start at 8K tokens per chunk and benchmark against 4K and 16K. For NVLink (intra-node), start at 8K as well; the optimal may be higher given the bandwidth headroom.

5. Start with TP=max_kv_heads, then scale CP. For an 8-KV-head model, TP=8 exhausts the tensor-parallel dimension. Scale context length by adding CP on top.

6. Benchmark TTFT vs throughput separately. Single-request TTFT is dominated by prefill time, which scales with context length. Throughput (tokens/sec aggregate across a request batch) is dominated by decode, which scales with batch size and KV read bandwidth. Optimize for the metric that matters for your workload.

Sequence parallelism for 4M-10M token contexts requires the right interconnect, not just more GPUs. Spheron's multi-node B200 and H200 clusters come with InfiniBand NDR fabric and transparent per-hour pricing, so you can model exact cost-per-million-tokens before committing to a long run.

Rent H200 for long-context inference → | Rent B200 multi-node → | View all GPU pricing →