GPU Utilization Training Bottleneck: Why Our GPUs Sat Idle

We had a training run underweight on throughput for three weeks and a procurement request half-drafted for a second GPU node before anyone ran nvidia-smi dmon next to htop. The fix that actually worked cost nothing: four lines in a PyTorch DataLoader call. This is the postmortem, with the numbers from that run.

TL;DR: GPU Utilization Training Bottleneck

A GPU utilization training bottleneck is often a starved dataloader, not undersized compute, and the two look identical on a dashboard showing only average GPU%.

• The pattern: GPU compute utilization averaged 44% while one CPU core sat pinned near 100%, the signature of a CPU-bound dataloader, not a GPU-bound model.
• The diagnostic: nvidia-smi dmon next to per-core CPU monitoring for two minutes confirms it, no profiler required.
• The fix: raising num_workers and prefetch_factor, plus pin_memory=True, with zero new GPUs bought.
• The buying consequence: the real upgrade was an instance with more vCPUs per GPU. Spheron varies this ratio by listing; compare current instance specs first.

GPU Utilization vs CPU Utilization: Reading the Signal Before You Blame the Hardware

GPU utilization and CPU utilization answer different questions, and reading either one alone tells you almost nothing about where a training run's real bottleneck sits. Here's how to read the pair together before you conclude anything about hardware:

• 100% GPU utilization does not guarantee there's no bottleneck. nvidia-smi's GPU-Util field reports whether any kernel was executing during the sample window, not how efficiently it used the streaming multiprocessors. A GPU can read 100% while running memory-bandwidth-bound kernels that leave most of the compute cores idle.
• 95% average GPU utilization is normal, not a red flag. For training workloads, 85-95% during active compute phases is close to the realistic ceiling once optimizer steps and gradient synchronization are accounted for. This is not the pattern we're describing in this post.
• 40% aggregate CPU usage next to 96% GPU usage is usually fine, not a bottleneck. But check per-core, not the aggregate: a 32-core box running four single-threaded dataloader workers pinned at 100% each will show roughly 12-13% aggregate CPU load while still being completely data-starved, because 28 idle cores drag the average down.
• The real tell is a periodic drop to near-0% GPU-Util, not a low average. A dataloader bottleneck doesn't show up as a uniformly lower number; it shows up as the GPU finishing a batch, then sitting idle for a fixed stretch every single step while the next batch is assembled.
• High GPU utilization with disappointing tokens-per-second can mean the opposite problem. If GPU-Util is consistently high with no periodic drops but throughput is still low, the GPU is more likely compute- or memory-bandwidth-bound, which is a different diagnosis with a different fix than anything in this post.

The Tell: 44% GPU Utilization While a Single Core Sat Pinned Near 100%

Our starting assumption was ordinary: throughput was below what the model size and batch size should have delivered, so the instinct was to add compute. That instinct is the default because it's usually correct for genuinely compute-bound jobs, and it's wrong often enough that it's worth checking before you spend on it.

As the Yotta Labs engineering team puts it, "When GPU utilization is lower than expected, the right question is not only 'which GPU are we using?' The better question is 'what is the GPU waiting on?'" That reframing is the entire fix. We were asking "do we have enough GPU," when the real question was "what is this GPU doing between batches."

This pattern is common enough to have a documented range. Our own number, 44%, sat right inside that band, which is what made the diagnosis fast once we knew what to look for.

This is not a storage problem. If your bottleneck is disk read throughput on a shared filesystem rather than CPU-side decode and augmentation, the diagnostic below will show a different pattern (steady GPU starvation with low CPU load, not a pinned core), and the fix is different too. Our guide to GPU Direct Storage covers that adjacent case.

Finding the GPU Utilization Training Bottleneck: nvidia-smi dmon Against the Dataloader Workers

The diagnostic takes about two minutes and needs nothing beyond tools that already ship with the driver and the OS. Run these two commands in separate terminals against a live training job:

nvidia-smi dmon -s u -d 1

htop

dmon with -s u prints one line per second per GPU showing SM utilization, memory utilization, and encoder/decoder usage. For the full flag reference, including power, clocks, and temperature columns, our GPU monitoring guide covers every dmon field in depth; this section only needs the utilization column.

Reading dmon Side by Side With CPU Load

Watch the sm column in the dmon output against htop sorted by per-core load, not the aggregate CPU percentage at the top of the screen. The pattern that confirms a dataloader bottleneck looks like this:

In our case, the sm column read single digits for roughly half of every second-long sampling window, on a cadence that matched the batch size exactly. htop sorted by core showed one core solid at 100% the entire time; the other 27 vCPUs on that instance sat mostly idle. That combination, not either metric alone, is what confirmed the diagnosis before we touched a single line of training code.

Why This Happens: What a Dataloader Worker Actually Does Between Batches

A DataLoader worker isn't idle time waiting on disk. Between handing off one batch and the next, it reads raw samples from storage, decodes them (image decompression, tokenization, or both), applies augmentations, and collates the results into a tensor the GPU can consume. All of that runs on CPU, in Python, and by default on a single process.

The mechanical reason this shows up as GPU idle time rather than a gradual, evenly distributed slowdown is how PyTorch's DataLoader assembles a batch. It constructs each batch synchronously, so if even one sample in that batch is slow to decode or augment, the entire batch waits on it before it can move to the GPU. This head-of-line blocking is why the failure mode looks like periodic cliffs in GPU utilization rather than a smooth, model-sized ceiling: the GPU finishes a step almost instantly, then stalls completely until the slowest sample in the next batch clears the pipeline.

With num_workers=0 or a low worker count, that CPU-side work runs on the same process (or a small number of processes) that also has to keep feeding the training loop, so there's no overlap between "prepare batch N+1" and "GPU computes batch N." Every additional worker process is another parallel preprocessing pipeline that can prepare a future batch while the GPU is busy with the current one, which is the entire mechanism behind the fix in the next section.

The Fix: num_workers, Prefetch Factor, and Pinned Memory

The change that closed most of the gap in our run was four DataLoader arguments, no new hardware:

train_loader = DataLoader(
    dataset,
    batch_size=64,
    num_workers=12,
    prefetch_factor=4,
    pin_memory=True,
    persistent_workers=True,
)

What each setting does, and why the defaults undersell it:

• num_workers: spawns that many separate processes to prepare batches in parallel with training. The default is 0, meaning no parallelism at all. Start near the number of physical CPU cores available to the job, not the vCPU count (which typically includes hyperthreads and overstates real parallel capacity), then adjust while watching dmon.
• prefetch_factor: controls how many batches each worker prepares ahead of time. The default of 2 is conservative; raising it to 4-6 gives workers more runway to stay ahead of the GPU, at the cost of a bit more host RAM.
• pin_memory=True: allocates batch tensors in page-locked host memory, which makes the CPU-to-GPU copy over PCIe faster and lets it overlap with compute instead of blocking on a pageable-memory transfer.
• persistent_workers=True: keeps worker processes alive between epochs instead of tearing them down and respawning them, which removes a startup tax that otherwise repeats every epoch.

None of these fixed a bug. They gave the CPU side of the pipeline enough parallelism to stay ahead of a GPU that had been finishing its work and then waiting.

When Tuning Isn't Enough: Offloading Preprocessing to DALI or WebDataset

If you've raised num_workers toward your physical core count and the GPU is still stalling, the preprocessing itself may be too heavy for CPU decode to keep up regardless of parallelism, common with high-resolution images, video, or complex augmentation pipelines. At that point the fix moves from tuning to architecture: shift decode and augmentation off the CPU entirely.

WebDataset takes a different angle at a related problem, streaming sequential shards instead of many small files, which helps when the bottleneck is filesystem metadata overhead rather than decode compute. It is worth reaching for only after the DataLoader-level tuning above has been tried and measured, since it adds real integration work that a four-argument change doesn't.

Right-Sizing vCPUs Instead of Renting More GPUs

The buying decision this postmortem changed wasn't "which GPU." It was "how many vCPUs does that GPU come with." The same mistake that shows up as a dataloader bottleneck in code shows up as a procurement mistake when the instance you rent doesn't have enough CPU parallelism to feed the GPU you paid for, regardless of how good that GPU is. This is the same lesson our GPU count vs training speedup piece makes about interconnect and communication overhead: adding more of the expensive resource doesn't fix a bottleneck that lives somewhere else in the stack, whether that's the network between GPUs or the CPU cores feeding a single one.

vCPU-to-GPU ratio varies meaningfully across instance listings, even on the same provider. On Spheron's marketplace, an H100 instance ships with fewer vCPUs behind it than an A100 instance does, since bundles are configured per listing rather than as one fixed CPU-to-GPU rule. If your workload is genuinely CPU-preprocessing-bound at moderate parallelism, an instance with a higher vCPU count per GPU is worth checking before assuming a second GPU is the answer, and each listing's full root access means you can actually run the dmon-and-htop diagnostic above yourself on the instance you're renting, rather than relying on a managed platform's own dashboard.

Two honest limits worth stating here. First, the pricing page shows preset bundles per listing, not an independent dial to raise vCPU count on a fixed GPU SKU; assembling a custom ratio outside those presets goes through the quote-based Custom Clusters path for larger deployments, listed as "8 to 512+ GPUs, specific hardware, InfiniBand configs on request," not a self-serve slider. Second, none of this diagnoses single-thread CPU clock speed, only core count and utilization pattern; a listing's published vCPU count tells you how much parallelism you can throw at a dataloader, not how fast any one of those cores runs, and that's a real gap if your bottleneck turns out to be per-sample decode speed rather than parallelism. Instance specs and pricing change with availability, so treat any ratio here as a snapshot, not a fixed rule.

If the underlying question is less "which instance" and more "how much infrastructure should this team be carrying at all," our guide to planning GPU capacity covers the sourcing side of that decision, and the GPU cost optimization playbook covers what a rented-but-idle GPU actually costs a training budget over a full run. In our case, the second GPU we'd nearly bought would have shown the identical 44% utilization pattern, on a second, equally starved instance.

Renting the wrong vCPU-to-GPU ratio is a silent cost, not a visible failure: the job still runs, just at half the throughput you paid for. Check the vCPU count behind each GPU listing before you assume the fix is another card.

Check H100 availability on Spheron →