GPU Direct Storage on GPU Cloud: Faster AI Training Checkpoints and Inference Loading (2026 Guide)

A 70B BF16 checkpoint is 140 GB. Through the standard CPU-staged path, writing that to NVMe takes 4-5 minutes per save. Run 1,000 checkpoints over a week-long training job and you waste 67-80 hours of GPU time sitting idle waiting on storage. Multiply that by $4.06/hr on-demand per H100 across an 8-GPU node and the wasted checkpoint I/O time alone costs over $2,100. GPU Direct Storage fixes this by letting the GPU's DMA engine write directly to the NVMe SSD, cutting that same 140 GB checkpoint to under 40 seconds. For teams building checkpoint resilience strategies on spot instances, GDS closes the gap between "theoretical fast recovery" and recovery that actually stays inside a 30-second preemption window.

The CPU Bottleneck GDS Eliminates

The traditional I/O path for a checkpoint write has five hops:

Hops 2 through 4 are pure overhead. The CPU copies data it will never compute on, just to stage it for the NVMe driver. For a 140 GB checkpoint, the CPU handles roughly 280 GB of memory movement total: 140 GB read from GPU and 140 GB written to NVMe. This is the bounce buffer problem. On a CPU with 63 GB/s peak DDR5 bandwidth that is also running DataLoader workers, NCCL communication, and the OS scheduler, that 280 GB of buffer traffic competes for memory bandwidth with every other process on the node.

The actual write to NVMe tops out at 7-14 GB/s depending on whether you have one or several PCIe Gen4/Gen5 drives. The bottleneck is not just the NVMe; it is every hop between GPU HBM and the SSD controller.

GDS Architecture: DMA Path via cuFile and nvidia-fs

GDS collapses hops 2 through 4 into zero. The data path becomes: GPU HBM -> PCIe bus -> NVMe controller. No CPU, no pinned buffer, no kernel copy.

Two components make this work.

nvidia-fs is a kernel-mode driver that registers with the NVMe driver to expose a P2PDMA (peer-to-peer DMA) interface. When a cuFile write call arrives, nvidia-fs coordinates with the GPU driver to initiate a DMA transfer directly from GPU VRAM onto the PCIe bus and into the NVMe controller queue, without involving any CPU cores or DDR.

cuFile is a userspace C library (also available as Python bindings via cuda-python). Applications replace pwrite(fd, buf, count, offset) with cuFileWrite(handle, buf, count, 0, offset). The OS sees a normal file descriptor. cuFile routes the call through nvidia-fs to the DMA engine.

The path, simplified:

One critical detail: GDS bypasses the Linux page cache entirely. Data written via cuFileWrite never lands in CPU DRAM. This is correct for checkpoint workloads where you write once and read once (on resume), but it means you cannot use mmap or page-cache-backed reads on GDS-written files. For read-heavy dataset pipelines, this tradeoff requires explicit planning.

GDS is part of NVIDIA's Magnum IO stack, which is the umbrella that covers GPU Direct Storage, GPUDirect RDMA, NCCL, and other direct data path technologies. Do not use "Magnum IO" and "GDS" interchangeably: Magnum IO is the product family, GDS is one specific technology within it.

GDS on CUDA 12.3+: Hardware Support Matrix

CUDA 12.3 expanded GDS to cover Hopper's DMA engine capabilities and PCIe Gen5 throughput improvements. There is no CUDA 13 as of this writing; GDS on CUDA 12.3+ covers the full Hopper and Blackwell GPU lineup.

The 250 GB/s figure for B200 is the Magnum IO sustained aggregate across an NVMe-oF fabric with multiple drives and RDMA interconnect, not a single-drive sequential write rate. A single PCIe Gen5 NVMe drive peaks at ~14 GB/s. The aggregate figure only applies when running a full NVMe-oF setup with InfiniBand or RoCE connecting multiple storage nodes to the compute cluster.

OS and software requirements:

Performance: Checkpoint and Weight Load Benchmarks

These numbers reflect single-node testing on H100 SXM5 with PCIe Gen4 NVMe. Multi-node figures use 8x H200 SXM5 with 4x PCIe Gen5 NVMe per node.

The 15% training throughput uplift sometimes cited for GDS comes from eliminating CPU I/O stalls in the DataLoader prefetch pipeline. When the data loading pipeline is bandwidth-bound at the CPU staging step, removing that bottleneck pushes more tokens per second through the training loop. This only applies to data-loading-bound workloads. Compute-bound jobs (pure forward/backward pass, not waiting on I/O) see no throughput gain.

Setting Up GDS on a GPU Cloud Node

These steps assume Ubuntu 22.04 with an NVIDIA Hopper or Blackwell GPU and at least one PCIe Gen4+ NVMe drive.

# 1. Verify NVMe drives are present
nvme list
lsblk | grep nvme

# 2. Install MOFED
wget https://content.mellanox.com/ofed/MLNX_OFED-<version>/MLNX_OFED_LINUX-<version>-ubuntu22.04-x86_64.tgz
tar -xvf MLNX_OFED_LINUX-*.tgz
cd MLNX_OFED_LINUX-*/
./mlnxofedinstall --force

# 3. Install nvidia-fs kernel module
apt-get install -y nvidia-fs-dkms
modprobe nvidia-fs
lsmod | grep nvidia_fs  # should show nvidia_fs

# 4. Verify GDS installation
gds_install_check
# Expected output: "GDS installation is verified"

# 5. Configure cuFile
cat > /etc/cufile.json << 'EOF'
{
  "logging": {"dir": "/tmp"},
  "profile": {"nvtx": false},
  "execution": {
    "max_io_queue_depth": 128,
    "max_batch_io_size": 131072,
    "allow_compat_mode": false
  },
  "properties": {
    "max_device_cache_size": 134217728,
    "max_pinned_memory_size": 67108864
  }
}
EOF

# 6. Test with a simple Python write
python3 -c "
import cupy as cp
import kvikio
a = cp.arange(1000000, dtype='float32')
with kvikio.CuFile('test_gds.bin', 'w') as f:
    n = f.write(a)
print(f'Wrote {n} bytes via GDS')
"

allow_compat_mode: false is the single most important config line. Without it, cuFile silently falls back to the CPU staging path on any misconfiguration, including a missing nvidia-fs module, wrong driver version, or unsupported file system. If compat mode is enabled and something is wrong, your benchmark will show the CPU path speeds and you will never know GDS was never active.

After the test write, confirm GDS activity at runtime:

cat /proc/driver/nvidia-fs/stats
# reads/writes counters should increment during GDS operations

kvikio (part of the RAPIDS ecosystem) is the recommended high-level Python interface for GDS. The raw cuda-python cuFile bindings are lower-level and better suited for custom FSDP writers. For most training use cases, kvikio is the right choice.

Training Use Case: DeepSpeed Checkpoint Offload with GDS

DeepSpeed ZeRO-3 with GDS

DeepSpeed's Async I/O module supports GDS natively via the aio config block. This is the lowest-friction path to GDS-enabled checkpointing.

{
  "zero_optimization": {
    "stage": 3,
    "offload_optimizer": {
      "device": "nvme",
      "nvme_path": "/mnt/nvme0/optimizer"
    }
  },
  "aio": {
    "use_gds": true,
    "block_size": 1048576,
    "queue_depth": 8,
    "thread_count": 1,
    "single_submit": false,
    "overlap_events": true
  }
}

Field notes:

• block_size: 1048576: 1 MB I/O chunk size, tuned for large sequential checkpoint writes. Smaller values increase kernel overhead with no throughput benefit for checkpoint workloads.
• queue_depth: 8: concurrent NVMe I/O operations submitted before waiting for completion. Eight is the right starting point; increase to 16 if you have multiple NVMe drives and are I/O-bound.
• thread_count: 1: set to 1 when using GDS to avoid contention between DeepSpeed's internal I/O threads and the DMA engine. Multiple threads do not improve GDS throughput and can cause lock contention in nvidia-fs.

Version note on use_gds: The aio.use_gds JSON config block applies to DeepSpeed's built-in async I/O path for optimizer offloading. In recent DeepNVMe documentation, GDS is also configurable via the aio_handle constructor when using the AsyncIOBuilder operator directly. If use_gds has no effect on your DeepSpeed version, check the DeepSpeed DeepNVMe documentation for your specific release and enable GDS at the handle creation level instead.

With ZeRO-3 on an 8x H200 cluster training a 70B model, optimizer states are roughly 560 GB total. Each rank holds a ~70 GB shard. With GDS, each rank writes that shard directly to its local NVMe in under 5 seconds. Without GDS, the same write takes 50+ seconds through the CPU staging buffer.

FSDP with GDS

PyTorch's torch.distributed.checkpoint.FileSystemWriter uses standard pwrite, not cuFileWrite. To route FSDP checkpoint writes through GDS, you need a custom StorageWriter backed by kvikio's CuFile.pwrite().

The pattern:

import torch
import torch.distributed.checkpoint as dcp
import kvikio

class GDSStorageWriter(dcp.StorageWriter):
    def __init__(self, path: str):
        self.path = path

    def write_data(self, plan, planner):
        for write_item in plan.items:
            tensor = planner.resolve_data(write_item)
            filepath = f"{self.path}/{write_item.storage_index.fqn}"
            with kvikio.CuFile(filepath, "w") as f:
                f.write(tensor.contiguous())

    def finish(self, metadata, results):
        # Only rank 0 writes metadata to prevent a race condition: finish() is
        # called by every distributed rank, so an unguarded torch.save() causes
        # all ranks to simultaneously write the same file, corrupting it.
        if not torch.distributed.is_initialized() or torch.distributed.get_rank() == 0:
            torch.save(metadata, f"{self.path}/.metadata")

For full FSDP2 sharded checkpoint support, the writer needs to handle per-rank sharding and distributed coordination. The pattern above shows the core GDS write call; production implementations should inherit StorageWriter fully and handle the write_metadata method.

For teams already using DeepSpeed for the broader training setup, the deepspeed.runtime.zero checkpoint utilities have built-in GDS support via the aio block and require no custom writers. For a full walkthrough of multi-node FSDP and DeepSpeed setup before reaching the checkpoint layer, that guide covers the full parallelism configuration.

Inference Use Case: Model Weight Loading and KV Cache Spillover

Weight Loading at Startup

Loading a large model at inference startup is the inference-side equivalent of writing a checkpoint: you are moving hundreds of gigabytes from NVMe into GPU VRAM. Without GDS, the path is NVMe -> CPU -> PCIe -> GPU. With GDS, it is NVMe -> PCIe -> GPU.

For Llama 4 Scout (109B) or DeepSeek V3 (671B), this difference matters. Load time on an H100 SXM5 with a single Gen4 NVMe drops from ~3 minutes to ~26 seconds for Llama 4 Scout. DeepSeek V3 on an 8x H200 node goes from ~18 minutes to ~2 minutes.

Using kvikio:

import torch
import kvikio

def gds_load_tensor(path: str, device: str = "cuda:0") -> torch.Tensor:
    with kvikio.CuFile(path, "r") as f:
        size = f.size()
        # size is in bytes; float32 is 4 bytes per element
        buf = torch.empty(size // 4, dtype=torch.float32, device=device)
        f.read(buf)
    return buf

For production weight loading, you would wrap this in a shard-aware loader that reconstructs the full model state dict from per-rank checkpoint files, handling dtype conversion and device placement.

KV Cache Spillover Acceleration

When KV blocks are evicted from GPU HBM to NVMe during high-concurrency inference (the three-tier pattern covered in depth in the NVMe KV cache offloading guide), GDS accelerates the eviction write path. Instead of KV block -> CPU staging buffer -> NVMe, GDS does KV block -> NVMe directly.

This matters most when the eviction rate is high: many concurrent users with long contexts, all competing for the same GPU HBM. Each eviction write that goes through the CPU adds latency and competes for DDR bandwidth. With GDS, evictions are handled by the DMA engine without touching DDR or CPU cores.

The read path on cache hit works the same way: the KV block comes back from NVMe directly into GPU HBM without CPU involvement, which cuts the cold-block retrieval latency.

Object Storage and NVMe-oF for Dataset Loading at Scale

GDS works beyond local NVMe. With NVMe-oF (NVMe over Fabrics) and RDMA, a compute cluster can access a remote NVMe storage pool at near-local speeds. With InfiniBand or RoCE networking and a GDS-compatible storage backend (VAST Data, WekaFS, NVIDIA Magnum IO fabric), a 32-GPU cluster can achieve over 100 GB/s aggregate read bandwidth for dataset prefetching.

The practical point: at petabyte-scale pre-training, the dataset loading pipeline sometimes sets the training step time rather than GPU compute. GDS over NVMe-oF removes the CPU from the data path entirely across the whole cluster, not just per-node.

One important constraint: NVMe-oF with GDS requires RDMA. It does not work over standard TCP/Ethernet without RDMA support. An InfiniBand or RoCE network with GPUDirect RDMA-capable NICs is required. Standard 100GbE Ethernet without RDMA cannot use this path.

Cloud object stores like S3 and GCS do not support GDS directly. Data on S3 must be downloaded to local NVMe first, then accessed via GDS for the local read. The GDS benefit applies to the local read step, not the S3 download.

Before committing to a parallel file system overlay, it is worth enabling GDS on each node's local NVMe first. A single PCIe Gen5 NVMe drive at 14 GB/s feeding GPU memory directly via GDS often makes the parallel FS step unnecessary for checkpoint workloads. See the parallel file system setup for multi-node training guide if you need to go beyond per-node storage.

Spheron GDS-Ready Clusters vs AWS p5 and GCP A3

Bare-metal GPU instances on Spheron include local NVMe and full root access. That means nvidia-fs and the full GDS stack are a one-time setup, not a platform restriction. On AWS p5, GDS requires a custom AMI. On GCP A3 Mega, there is no local NVMe at all (storage is Hyperdisk, which does not support GDS).

For a 70B model (140 GB BF16 checkpoint), 1,000 checkpoints over a week-long run:

Pricing fluctuates based on GPU availability. The prices above are based on 06 Jun 2026 and may have changed. Check current GPU pricing → for live rates.

The gap between GDS and non-GDS on Spheron is roughly $1,871 per week-long run at on-demand rates. On AWS, that same checkpoint I/O waste costs roughly $3,688. The underlying reason is the same (CPU-staged I/O is slow), but the dollar impact scales with the hourly rate. Setting up GDS on Spheron's H100 instances takes one-time setup versus per-run cost. Spheron's spot pricing at $1.43/hr per H100 reduces these absolute figures further, while AWS p5.48xlarge spot rates vary.

For teams scaling to H200 or B200, the math shifts further. NVMe-backed H200 nodes from $5.55/hr on-demand per GPU (or $1.77/hr spot) and B200 from $8.61/hr on-demand per GPU (or $5.34/hr spot) with Blackwell's faster NVMe DMA engine further close the gap. The GDS setup steps are identical across all three GPU types. For the highest checkpoint throughput on a multi-node cluster, rent B200 GPU nodes support Magnum IO NVMe-oF and deliver 250 GB/s sustained across the fabric.

Spheron bare-metal H100, H200, and B200 nodes include local NVMe and full root access, which means nvidia-fs and GDS are a one-time setup rather than a platform restriction. Per-minute billing keeps checkpoint-heavy runs cost-efficient even for short experiments.

H100 on Spheron → | H200 SXM5 availability → | B200 for Blackwell GDS →