Bare Metal vs Virtualized GPU: Performance Consistency

Every GPU cloud lists an hourly rate for its virtualized instances that undercuts the bare-metal tier on the same page. What that rate doesn't show is what happens to your P99 time-to-first-token when another tenant on the same physical host starts a training run at minute 40 of your inference workload. A GPU cloud performance consistency benchmark has to look past the sticker price to the variance underneath it, because for a latency-SLA'd product, the tail is the bill.

TL;DR: Bare Metal vs Virtualized GPU Instances, What the Variance Actually Costs

• Isolation, not virtualization, drives the tax. A full-size, single-tenant AWS Nitro instance matches bare metal within 1% on HPC workloads.
• Sharing layers alone move TTFT by ~1.6x. GPU-Virt-Bench measured 45.2ms under HAMi-core versus 28.7ms under BUD-FCSP on identical hardware (percentile unstated).
• The market prices this in. CoreWeave's bare-metal architecture is tied to roughly a 10-15% per-GPU-hour premium over competitors.
• Spheron sells both tiers. Its H100 page lists bare-metal clusters up to 80x H100 alongside virtualized VM instances.
• Even clean bare metal has spread. Spheron's own dedicated H100 benchmark logged TTFT P50 at 78ms and P99 at 342ms, a 4.4x ratio with zero noisy-neighbor contention.

Hypothesis: The Sticker Price on a Virtualized GPU Instance Hides a Variance Tax

The hypothesis going in was simple: a virtualized, potentially shared GPU instance costs less per hour than its bare-metal equivalent, but that discount doesn't show up as a flat percentage on your invoice. It shows up as increased spread between your P50 and P99 latency, and if your product has a latency SLA, that spread is what you pay for later, in over-provisioned headroom, in SLA credits, or in the accounts you lose when the tail gets bad enough that someone notices.

This isn't a new idea in cloud infrastructure generally. TechTarget's cloud computing glossary lists bare-metal cloud as a direct mitigation for the noisy-neighbor effect, on the reasoning that putting one tenant on a physical server removes the shared-resource contention that causes it in the first place. What's less settled is how big that effect actually is on a GPU specifically, where the shared resource isn't just CPU cycles or disk I/O but VRAM bandwidth, PCIe lanes, and the GPU's own scheduling queue, none of which behave like a CPU core under a cgroup.

Test Setup: A GPU Cloud Performance Consistency Benchmark on Bare Metal and Virtualized Instances

Isolating "virtualization overhead" from every other variable in a live GPU cloud is harder than it sounds, because two providers running the same GPU model rarely run identical drivers, identical network fabric, or identical co-tenant load at the moment you test. Rather than compare two live providers where a dozen confounding variables move at once, the cleanest available evidence isolates the sharing mechanism as the single variable on identical hardware, which is what an academic benchmarking framework called GPU-Virt-Bench did: it ran the same synthetic transformer inference kernels under two different GPU-sharing layers, HAMi-core and a newer scheduler called BUD-FCSP, on the same underlying hardware, and measured time-to-first-token under each. The paper's own table reports these as raw TTFT figures without stating which percentile or aggregation (mean, median, P99) they represent, so treat the comparison as one measured run between two sharing layers, not a confirmed tail-latency statistic.

Alongside that, this post uses a real, dedicated (non-shared) H100 baseline from Spheron's own fleet: a prior benchmark ran 100 concurrent agent sessions against Llama 3.1 8B Instruct in FP8 on a single bare-metal H100 SXM5, with vLLM's chunked prefill enabled, and logged TTFT at P50, P95 and P99 client-side. That run had no co-tenant on the physical host by construction, so its P50-to-P99 spread is a useful floor: this is what tail latency looks like from scheduling and batching alone, before any noisy-neighbor tax gets added on top. This is also why a single throughput number is never enough to size an inference deployment: because a peak-throughput figure alone says nothing about how a GPU behaves under a latency budget.

GPU Cloud Performance Consistency Benchmark Results: TTFT Under Load

Two things stand out. First, even a genuinely clean, dedicated bare-metal instance carries real P50-to-P99 spread: 4.4x on Spheron's own agent workload, driven by request scheduling and prefill batching, not by any other tenant. If your SLA budget assumes P99 sits close to P50, that assumption is wrong even on hardware you don't share. Second, the GPU-Virt-Bench numbers show the sharing layer itself, not just the presence or absence of a co-tenant, moving TTFT by roughly 1.6x on identical hardware and an identical kernel, even though the paper doesn't say which percentile that 45.2ms-versus-28.7ms gap represents. That's a variance tax showing up purely from software architecture choices inside the virtualization stack, before a second tenant's actual workload is even part of the picture, though it should be read as one measured comparison rather than a confirmed tail-latency statistic.

Read together with the AWS Nitro figure, the pattern is: isolation removes the tax, virtualization technology by itself does not add one. A full-size, single-tenant Nitro VM tracks bare metal within 1%. A shared instance, on any virtualization stack, is the condition where the tax appears, and the GPU-Virt-Bench numbers show it can be a meaningfully sized tax even before a live noisy neighbor is running anything.

What Causes the Gap: Hypervisor Scheduling, vCPU Descheduling, and Memory Contention

Three mechanisms explain most of the spread between an isolated instance and a shared one:

• Hypervisor scheduling of vCPUs. On a shared host, your instance's vCPUs are descheduled to give CPU time to a neighbor's vCPUs. For a GPU inference server, this matters more than it sounds: request preprocessing, tokenization, KV cache bookkeeping and the Python-side scheduler loop in a framework like vLLM all run on CPU, and a descheduled vCPU stalls the request feeding the GPU, not just background work.
• GPU-level time-slicing or sharing-layer overhead. When a physical GPU is split across tenants (via MIG, MPS, or a software sharing layer), the mechanism that arbitrates between tenants adds its own latency, and different implementations add different amounts. That's exactly what the GPU-Virt-Bench numbers above isolate: two sharing layers, same hardware, a roughly 1.6x TTFT gap from the arbitration logic alone.
• Memory bandwidth contention. GPU memory bandwidth is a shared physical resource in any multi-tenant configuration, and a neighbor running a bandwidth-heavy kernel (a large batch matmul, a memory-bound attention pass) can measurably slow your own kernels without touching your compute allocation at all, since you're both waiting on the same memory bus.

None of these three mechanisms show up in a spec sheet. A shared instance and a dedicated instance can list identical GPU model, identical VRAM, identical driver version, and only diverge under concurrent load, which is exactly the condition a spec comparison never tests and a P99 benchmark does.

Not All Virtualization Is the Same Tax: Why AWS Nitro's ~1% Overhead Doesn't Contradict This

It's worth being precise about what the AWS Nitro result actually shows, because it's easy to misread as "virtualization has no cost." AWS's HPC blog benchmarked a full-size Nitro instance, c5n.18xlarge, against its bare-metal equivalent, c5n.metal, and found performance within 1% across WRF, GROMACS, OpenFOAM and HPL. That same blog post explains why the number lands there: it describes the Nitro hypervisor as having "only necessary functions, with a design goal of being quiescent, which means it should never activate unless it is doing work for an instance that requested it," so on a full-size instance with no other tenant, there's effectively nothing for the hypervisor to interfere with.

The 1% figure is a single-tenant result: a full-size instance with the whole physical host to itself. It confirms that a quiescent hypervisor adds negligible overhead when there's no contention to arbitrate, which is a different claim from "shared virtualized instances perform like bare metal," and AWS's HPC blog doesn't make the second claim. A smaller instance size on the same virtualization stack is, by definition, a different configuration: it shares a physical host with other tenants rather than getting the full machine, which is exactly the condition the 1% benchmark never ran.

What the Noisy-Neighbor Tax Costs Over a Month of Production Traffic

Put a number on it using the two real data points above. Spheron's own dedicated H100 baseline holds P99 TTFT at 342ms against an 800ms SLA budget, for a workload of 100 concurrent agent sessions. That's 458ms of headroom before an SLA breach. If a comparably loaded shared instance carries the same order-of-magnitude sharing-layer tax the GPU-Virt-Bench paper measured on its own TTFT figures (45.2ms versus 28.7ms, roughly a 1.6x gap, on an unstated percentile), that dedicated instance's 342ms P99 would need to survive becoming something closer to 550ms before a live noisy neighbor's own workload adds anything on top. Headroom against an 800ms budget drops from 458ms to roughly 250ms, a loss of more than 40% of your SLA margin from the sharing mechanism alone, before a second tenant's actual traffic pattern is even a factor. Treat this as an illustrative worked example built from a single measured comparison, not a guaranteed multiplier.

There are two ways to buy that margin back, and the market already prices both. One is to pay for isolation directly: CoreWeave's roughly 10-15% per-GPU-hour premium over competitors is, per the independent review that measured it, tied in part to exactly this bare-metal, no-hypervisor architecture. The other is to buy the margin back with raw capacity: over-provision enough extra GPU-hours of headroom on the shared tier that your effective concurrency, and therefore your queueing-driven P99, stays where it would have been on dedicated hardware. Provisioning roughly 60% more GPU-hours than your dedicated baseline required, to absorb a comparable P99 blowup, is a materially larger ongoing cost than a 10-15% hourly premium paid once. Whichever path you take, the "cheap" instance's true monthly cost includes the fix, not just the hourly rate advertised on the page.

When a Virtualized GPU Instance Is Still the Right Call

None of this makes a shared, virtualized instance the wrong choice for every workload. It makes it the wrong choice for the specific case of a latency-SLA'd, user-facing service running at sustained concurrency. Plenty of GPU workloads aren't that:

• Batch and offline jobs. A nightly embedding run, a document-classification pipeline, or an async summarization queue has no user waiting on TTFT. A P99 blowup that would breach a chat SLA is invisible to a job that just needs to finish before morning.
• Development and experimentation. Iterating on a fine-tuning run or testing a new serving config doesn't need SLA-grade consistency, and paying a bare-metal premium for that phase of work is money spent on a guarantee nobody's using yet.
• Bursty, low-average-utilization traffic. If your real bottleneck is provisioning speed and the ability to scale down to near-zero between bursts, a virtualized or serverless instance's faster spin-up and finer-grained sizing can matter more than tail-latency consistency on a workload that isn't sustained enough to expose the noisy-neighbor effect in the first place. Spheron's comparison against Runpod covers this trade-off directly, since Runpod's serverless containers optimize for exactly this kind of fast, bursty spin-up.
• Cost-constrained early-stage products. If your SLA is "we'll get back to you," not a contractual latency number, the discount on a shared instance is a real discount, not a deferred cost.

Spheron itself doesn't force this decision one way: its pricing page describes what a customer gets as "a fully provisioned VM or bare metal instance with your selected GPU," and its own docs separately pitch "bare metal performance" as part of a 60-80% cost-savings claim relative to hyperscalers. That range spans both instance types, which is the honest way to read a marketplace aggregating 5+ providers: it doesn't publish a single cross-provider TTFT variance number the way this post just built one, and it doesn't uniformly assign every listing to one instance type either, so the tier you land on depends on the specific listing, not the platform as a whole. For a production inference service where the SLA is real, that's the one line worth checking before you provision: which instance type, on that specific listing, you're actually renting. For more on what bare metal removes from the GPU path, no hypervisor overhead and dedicated VRAM, see Spheron's bare-metal vs serverless GPU comparison and the broader GPU cloud benchmark set this post's numbers sit alongside.

A P99 SLA is only as reliable as the isolation under it. If your workload has a real latency budget, check whether the instance you're pricing is dedicated or shared before you commit.

Rent a dedicated H100 on Spheron →