Same code, same H100, same batch size. Tokens per second on one node dropped by roughly 30% between a Monday deploy and a Wednesday one, and nothing in our serving stack had changed in that window. The only thing that had moved was the CUDA driver, pulled in by a routine OS package update over a maintenance window. This is what a GPU throughput regression after update looks like from the inside: the causes we ruled out first, the metric that actually caught it, the fleet-wide diff that pinned it to a single package version, and the pinning and gating practice we run on every node now.
TL;DR: What Causes a GPU Throughput Regression After Update?
- Cause: an unpinned CUDA driver update changed cuBLAS/cuDNN kernel selection and dropped CUDA graph replays to eager execution, cutting tokens/sec roughly 30% on the same H100.
- Known pattern: NVIDIA's TensorFlow container notes document a cuDNN regression cutting EfficientNet throughput up to 30% on H100, still open in the 23.11 notes.
- Metric that caught it:
DCGM_FI_PROF_PIPE_TENSOR_ACTIVEtracked the drop while plainnvidia-smiGPU utilization stayed flat. - Fix: pin the exact point version with
apt-mark hold, not just the major branch, and gate every driver promotion behind a canary tokens/sec benchmark. - Before you rent: Spheron lists full root access on VM and bare-metal tiers, the access a driver pin needs. See the kernel-development coverage.
The Symptom: A GPU Throughput Regression After Update
We run a production inference fleet on H100 SXM5 80GB nodes serving an FP8 Llama-class model behind vLLM. On a Wednesday morning, tokens per second on one node had fallen by roughly 30% from where it had sat for the previous six weeks, against the exact same container digest, the exact same --max-num-seqs setting, and the exact same request mix logged by the load balancer. No deploy had gone out for that service since the prior Friday.
What Didn't Change: Ruling Out the Model, Batch Size, and Thermals First
The instinct with a throughput drop is to blame the workload first, so that's where we started, and eliminated each candidate in turn:
- Model weights: the container image digest and model checksum matched the healthy baseline exactly. No re-deploy, no quantization change.
- Batch size and sequence settings:
--max-num-seqs,--max-model-len, and chunked prefill were unchanged in the vLLM launch config. - Prompt mix: request logs over the affected week showed the same input and output token distribution as the six prior weeks. Nothing had shifted client-side.
- Thermal state: GPU temperature and SM clock speed, pulled from
nvidia-smi -q, sat in the normal range with no throttling event recorded. This wasn't a power or cooling problem.
If you're chasing a similar mystery drop and haven't already ruled out kernel-level causes like warp divergence, that's a faster check to run before you start diffing driver versions across a fleet. It wasn't our answer here, but it's a cheaper one to rule out first.
The Metric That Caught It: DCGM_FI_PROF_PIPE_TENSOR_ACTIVE vs Plain GPU Utilization
nvidia-smi's headline GPU-Util number stayed exactly where it always had, high 80s to low 90s. That's the field most dashboards alert on, and it told us nothing was wrong. What moved was DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, the DCGM profiling counter that reports the fraction of time the tensor pipe itself is issuing work, not just the fraction of time some kernel occupies the SM. It tracked the tokens/sec drop almost exactly, while GPU-Util didn't move.
That gap between "the SM looks busy" and "the tensor pipe is actually doing the matrix-multiply work you're paying for" is worth separating clearly. We cover the difference between SM occupancy, which describes warp-scheduling headroom, and tensor pipe activity in more detail in our CUDA occupancy calculator piece and our breakdown of GPU spec sheets versus real-world tensor core utilization. This incident is what made the distinction matter in production rather than in a benchmark: a GPU can look fully busy on GPU-Util and still be quietly running a slower kernel path underneath it.
How We Traced the CUDA Driver Update Performance Regression
Diffing nvidia-smi Driver Versions Across the Fleet Before and After the Drop
With the model, batch settings, and thermal state cleared, the next step was comparing the regressed node against a healthy one running the identical container. nvidia-smi prints its driver version on its own header line, so the fastest check across a fleet is a one-line diff: pull nvidia-smi --query-gpu=driver_version --format=csv from every node, dedupe the output, and see whether more than one version is running behind the same image.
On our fleet it wasn't subtle. The regressed node was running a driver two point releases ahead of every other node in the pool. It had rolled through a maintenance window over the weekend that ran a routine OS package update, and the node's nvidia-driver package hadn't been held, so it moved along with everything else in that apt-get upgrade.
Why This Wasn't a One-Off: NVIDIA's Own Release Notes Document the Same Pattern
This isn't a theory we invented to explain one bad afternoon. NVIDIA's own TensorFlow container release notes carry an open known issue: "There is a known cuDNN performance regression that can reduce performance by up to 30% for the EfficientNet model on H100. This will be fixed in a future release." That line appears in the 23.11 release notes. An earlier and separately caused EfficientNet slowdown, up to 50% and attributed to upstream TensorFlow rather than cuDNN, was already marked "under investigation" back in the 23.02 release notes. This class of bug has recurred under different root causes across multiple version cycles, not just once.
It isn't confined to training frameworks either. NVIDIA's TensorRT 10.0.1 release notes document a driver-specific issue of their own: "There is a small chance that TensorRT will hang when running on H100 with the r550 CUDA driver when CUDA graphs are used," with the documented workaround to "use the r535 CUDA driver instead or avoid using CUDA graphs." A specific driver branch, paired with a specific optimization technique, producing a specific failure mode, is exactly the shape of bug we were chasing.
The Specific Mechanism We Found: Kernel Selection and CUDA Graph Behavior
Once the version diff pointed at the driver, we went to Nsight Compute to see what had actually changed at the kernel level rather than guess. The kernel names captured for the same attention and GEMM operations differed between the two driver versions: the regressed node was landing on a narrower-tile GEMM kernel for a subset of our sequence-length buckets, the kind of shift that shows up when a driver update changes which kernel a given shape gets routed to at the cuBLAS or cuDNN heuristic layer. We also saw a handful of CUDA graph replays fall back to eager execution for our longest-context requests, consistent with the graph-capture caveat NVIDIA documents against the r550 driver branch in the TensorRT notes above. Neither change touched a single line of our own code.
What We Changed: Pinning Versions and Catching Regressions Before Production
Pinning the Exact Point Version with apt-mark hold
NVIDIA's own forum staff give the direct fix for this class of problem. The detail that actually matters is which string you hold. Pinning the major branch alone, nvidia-driver-550, still lets apt-get upgrade move you to a newer point release inside that same branch, which is exactly the kind of change that shipped the regressions cited above. We now pull that exact string from dpkg -l | grep nvidia-driver on every node at provisioning time and hold it in the same step, instead of trusting a branch-level pin to stay put.
unattended-upgrades is the usual culprit behind this kind of drift, and it's worth saying plainly: as one troubleshooting guide to NVML version mismatches puts it, "the mismatch error is self-inflicted 90% of the time: unattended-upgrades pulling a new driver on a running box." That's from a write-up on fixing NVML driver/library version mismatches, and the fix for that specific failure mode is the same pin.
A Pre-Production Regression Gate: Baseline Tokens/Sec Before Promoting Any Driver
We also stopped treating a driver update as a decision made once, silently, at image-build time. Every candidate driver now runs through a fixed micro-benchmark, same prompt mix, same batch size, same sequence-length distribution as production, on one canary node before it's allowed onto the rest of the pool. The gate checks two numbers against that node's own rolling baseline: tokens/sec and DCGM_FI_PROF_PIPE_TENSOR_ACTIVE. A drop past a set threshold on either one blocks the promotion and pages whoever owns the fleet, instead of surfacing three days later as a mystery ticket.
This is the same discipline goodput engineering asks for everywhere else on a cluster: treat the number the business actually pays for as the thing you gate on, not the number the dashboard happens to default to showing. Reading vendor release notes as routine input, rather than after-the-fact explanation, is part of the same practice now too.
What to Check Before You Rent: Who Controls the Driver on This Instance
None of the fix above works without root. apt-mark hold needs to run as a user who can touch the package manager, and confirming the real installed version with nvidia-smi only means something if there's no hypervisor layer quietly presenting a different driver than the one you'd actually pin. Bare metal specifically removes the hypervisor layer, so nvidia-smi is reading the real installed driver rather than something a host layer could reinterpret.
Worth saying plainly: neither Spheron's pricing page nor its docs publish a specific default driver version, an update cadence, or an SLA on driver-version stability as of this writing, so the responsibility for pinning and testing before promoting a driver still sits with the tenant, the same as it does on the checklist above. As of 20 Sep 2026, Spheron lists H100 SXM5 at $2.98/hr on-demand and $2.10/hr on spot, both tiers carrying the same root-access terms.
Pricing fluctuates based on GPU availability. Spheron rates above are live as of 20 Sep 2026; other providers reflect their most recent published rates and may have changed. Check current GPU pricing → for live rates.
If you're on a managed platform that controls its own base image instead of handing you root, the right move is different: ask that provider directly what its driver-update and rollback policy is, because you can't run the pin yourself the way a root-access renter can.
When a Driver Update Is Still Worth Taking
The lesson here isn't "never touch the driver." The exact release cited above for the inter-die traffic fix, 570.172.08, shipped specifically to fix a performance regression that an earlier driver on the same branch had introduced. Security patches land in driver updates too, and new GPU generations sometimes require a driver floor the old pinned version doesn't meet. Holding a version forever just relocates the risk from "silent regression" to "known issue sitting unpatched."
What changes is the process, not the decision to ever update. Every candidate driver goes through the same canary and gate described above before it touches the rest of the fleet, and whatever version it lands on gets the exact same apt-mark hold treatment the day it's approved. A driver update stops being something that happens to you silently between two deploys and becomes a release you promote on purpose, with a number attached to whether it's allowed to ship.
If a routine driver update can cost you 30% of the tokens/sec you're already paying for, the fix starts with an instance where you can actually see, and pin, what's installed. Spheron rents H100 GPUs with full root access on both VM and bare-metal tiers, so the pin is yours to set, not a ticket you file.
Frequently Asked Questions
nvidia-smi's headline GPU-Util field counts the fraction of time any kernel occupied an SM, which stays high even when a workload has fallen back to a slower kernel path. DCGM_FI_PROF_PIPE_TENSOR_ACTIVE is a DCGM profiling counter that isolates the fraction of time the tensor core pipe specifically was issuing work. A GPU throughput regression after a driver update can leave nvidia-smi utilization unchanged while DCGM_FI_PROF_PIPE_TENSOR_ACTIVE drops, because the SM is still busy, just not on tensor core math.
Run apt-mark hold against the exact package string, not just the major branch. Pull the installed version with dpkg -l | grep nvidia-driver, then hold that full epoch and point-version string (something like 550.127-1~ubuntu22.04), since holding only nvidia-driver-550 still lets apt-get upgrade move you to a newer point release inside that same branch.
Yes. NVIDIA's own TensorFlow container release notes document a cuDNN performance regression that reduces EfficientNet throughput by up to 30% on H100, an issue that stayed open across the 23.07 and 23.11 release notes with no code change required to trigger it. NVIDIA's TensorRT 10.0.1 notes separately document a CUDA graph hang risk tied specifically to the r550 driver branch.
Spheron's pricing page describes its hourly rate as covering a fully provisioned VM or bare metal instance with full root access, which is what lets a tenant run apt-mark hold and confirm the real driver version with nvidia-smi. Neither the pricing page nor Spheron's docs publish a default driver version or an update cadence, so pinning and pre-promotion testing is still the tenant's responsibility, the same as on any other root-access provider.
