NVIDIA H100 vs RTX 4090 for AI: Data Center vs Consumer GPU Compared (2026)

A team training a 7B model on an RTX 4090 hits a wall when they want to run the same job on a 70B model or scale to multi-GPU. The question becomes: rent an H100 for the training run, or buy another 4090? This post gives you the numbers to make that call, covering VRAM ceilings, inference throughput, cost per million tokens, and the training scenarios where the 4090 simply cannot compete.

Short answer: the 4090 wins for development, sub-13B inference, and cost-sensitive serving on smaller models. The H100 wins for 70B+ workloads, production serving, and anything requiring FP8 or NVLink.

Quick Answer

Specs Side by Side

Pricing as of 25 May 2026. Check current GPU pricing → for live rates.

One clarification on FP8: the RTX 4090 has FP8 Tensor Core support (hence 1,321 AI TOPS), but it does not have NVIDIA's Transformer Engine. That is a hardware-plus-software pipeline specific to Hopper that automatically manages FP8 scaling factors across attention and feed-forward layers. The H100 reaches 3,958 TFLOPS on actual transformer operations through this pipeline. The 4090's FP8 is raw Tensor Core arithmetic without the engine that makes it practical for LLM workloads.

Also worth noting: the RTX 4090 has no NVLink whatsoever. The RTX 3090 had an NVLink bridge option, but NVIDIA removed it entirely on the 4090. Multi-GPU on a 4090 cluster is PCIe Gen 4 only.

What Fits: VRAM Ceiling by Model

For Llama 3.3 70B at Q4_K_M: model weights need roughly 35 GB (70B * 0.5 bytes average = 35 GB). This exceeds the 4090's 24 GB completely. On the H100's 80 GB, it fits with 45 GB to spare for the KV cache. On the 4090, even two cards (48 GB combined via PCIe) can technically load it, but the PCIe bandwidth makes tensor parallelism impractical for real throughput.

For Qwen3 32B on the 4090: the Q4 weights (~20 GB) fit in 24 GB, but vLLM's default context lengths add several GB of KV cache on top, pushing past the limit. You can set --max-model-len 2048 to work around this, which is fine for short-context tasks but not practical for production serving. For the full model capacity matrix across GPUs, see GPU memory requirements for LLMs.

Training: Where the 4090 Cannot Go

For anything that fits in 24 GB, the 4090 is a capable fine-tuning GPU. QLoRA on models up to 20B, LoRA on models up to 13B, and full fine-tuning on models up to 3B all work fine. The RTX 4090 for AI/ML full specs and benchmarks covers these in detail.

The hard limits show up in three places.

NVLink absence. PCIe Gen 4 gives 32 GB/s per direction between GPUs. The H100's NVLink 4.0 gives 900 GB/s total bidirectional bandwidth. In an 8-GPU training cluster, the all-reduce operations that synchronize gradients across GPUs are bandwidth-limited. A 4090 cluster can do data parallelism (send gradients, not activations), but tensor parallelism and pipeline parallelism require bandwidth that PCIe cannot provide at scale. The H100's NVLink fabric is what makes 8-GPU tensor parallelism practical on 70B+ models.

No FP8 Transformer Engine. For training with FP8, the H100 reaches 3,958 TFLOPS on attention and feed-forward layers. The 4090 does not have this pipeline, so BF16 (165.2 TFLOPS) is its practical ceiling for training. For a detailed explanation of FP8 and the Transformer Engine, see our FP8 quantization guide.

Multi-node scaling. The H100 SXM5 form factor connects to NVSwitch via NVLink and supports InfiniBand for inter-node communication. The 4090 is single-node PCIe only. If your training job needs more than one physical machine, you need data center hardware.

Training throughput comparison for representative workloads:

Training benchmarks for 70B on 8x H100 from published MLPerf training v4.0 results and community benchmarks at rank 64 LoRA, 4K sequence length, gradient checkpointing. Single-GPU QLoRA estimates based on memory bandwidth ratios.

Inference: Where the 4090 Punches Above Its Weight

For single-user inference on 7B-13B models, the 4090's 1,008 GB/s bandwidth is genuinely competitive. Running Llama 8B Q4_K_M with llama.cpp or Ollama:

For personal/dev use on sub-13B models, 80-120 tok/s is interactive speed. Nobody needs 400 tok/s to iterate on prompts. The H100 is faster, but the 4090 at $0.55/hr is more economical for solo development.

The picture changes for multi-user serving with vLLM. At high batch sizes, the H100's bandwidth advantage compounds and its higher compute TFLOPS become relevant for prefill operations:

For serving Llama 8B at scale, the cost-per-token picture depends on the hourly rate. See the next section for the full math.

The 4090's bandwidth (1,008 GB/s) is genuinely strong for a $0.55/hr GPU: it is the same bandwidth as a server-grade PCIe GPU that cost $8,000 a few years ago. For small-model dev inference, it holds its own.

Multi-GPU Reality: 8x 4090 vs 1x H100 vs 8x H100

When teams outgrow a single GPU, the choice gets more interesting.

8x H100 SXM5 monthly estimate based on $30.75/hr cluster rate (cheapest 8-GPU offer at time of writing). 8x RTX 4090 at $0.55/hr x 8 = $4.40/hr.

Eight RTX 4090s via PCIe can technically load a 70B model across cards (via tensor parallelism or pipeline parallelism), but the 32 GB/s PCIe bottleneck makes tensor parallelism slow. What 8x 4090 does well: data-parallel inference of independent requests on sub-24B models, where each card handles separate requests without cross-GPU communication. For training workloads requiring gradient synchronization at scale, PCIe bandwidth is the bottleneck and performance falls significantly below NVLink clusters.

A single H100 SXM5 covers most teams: it handles 70B Q4 inference on a single card and provides ECC memory and MIG partitioning that 8x 4090 cannot. The 8x H100 cluster is for training 70B+ models in BF16 or FP8, or serving 405B+ models in production.

Cost Per Million Tokens: The Real Math

Using live Spheron on-demand pricing as of 25 May 2026.

Formula: Cost/M tokens = hourly_rate / (tokens_per_second x 3,600) x 1,000,000

Llama 3.1 8B, batched vLLM:

At current on-demand prices, the RTX 4090 is cheaper per token for Llama 8B, even against the H100 at full batch throughput. The H100 FP8 narrows the gap to $0.089/M tokens but still costs more than the 4090. The H100's value shifts to workloads the 4090 cannot handle: models above 24 GB VRAM, FP8 training with the Transformer Engine, and high-concurrency production serving where its MIG and ECC features matter.

Llama 3.3 70B Q4, batched vLLM (single GPU):

Owned RTX 4090 depreciation math:

• Hardware: $2,000 system (GPU + workstation) over 3 years = $0.076/hr in depreciation
• Electricity: $0.15/kWh x 450W = $0.068/hr at full load (assumes $0.15/kWh, adjust for your region)
• Total effective cost at 100% utilization: ~$0.144/hr

At 2,550 tok/s and $0.144/hr: $0.016/M tokens. Owned 4090 beats every rental option at 100% utilization. At 50% utilization (more realistic): fixed costs stay the same while billable hours halve, so effective rate is ~$0.288/hr = $0.031/M tokens. Still competitive. The break-even point against H100 rental depends on your actual utilization and the specific task.

Pricing fluctuates based on GPU availability. The prices above are based on 25 May 2026 and may have changed. Check current GPU pricing → for live rates.

For a full cost-per-token comparison across more GPU models and quantization levels, see GPU cost-per-token benchmarks for LLM inference 2026.

When the 4090 Wins

• Solo experimentation and prototyping on sub-13B models
• Single-user or low-concurrency inference at $0.060/M tokens, the cheapest on-demand rental option for sub-13B models
• QLoRA fine-tuning up to 20B parameters
• Hobbyist and researcher workloads where the priority is low hourly cost, not maximum throughput
• Always-on local development environments where owned hardware makes sense at high daily utilization

When the H100 Wins

• Training runs on 30B+ models, especially with FP8 via the Transformer Engine
• 70B+ inference in production: the H100 is the smallest single GPU that comfortably runs Llama 3.3 70B at Q4
• Production multi-user serving with ECC memory and MIG partitioning for multi-tenancy
• Multi-GPU clusters where NVLink bandwidth enables tensor parallelism
• Short-burst training runs: renting H100 on-demand only when you need it means no depreciation, no electricity costs, and no hardware commitment

For a detailed breakdown of the H100 SXM5 architecture including all precision tiers, see the H100 complete datasheet. For how the H100 compares to its predecessor, see the A100 vs H100 comparison.

Hybrid Play: 4090 for Dev, Spheron H100 for Training

The most cost-efficient pattern for small teams: keep a 4090 for daily development and fast iteration on sub-13B models, then rent H100 SXM5 on Spheron only for full fine-tuning runs.

Example monthly cost breakdown for a 3-person AI team:

Compare that to renting an H100 full-time for all work (including idle dev hours):

• $3.84 x 24 hrs x 30 days = $2,764.80/mo

The hybrid approach cuts cost by 94% while still giving you H100 capability when the run actually needs it. For the fine-tuning setup itself, see our complete LLM fine-tuning guide.

You can rent H100 on Spheron on-demand with per-minute billing, meaning you only pay for the hours the training job actually runs. Current H100 pricing applies.

Training Runs: A Worked Example

A typical LoRA fine-tuning run on Llama 3.1 70B takes approximately 8-12 hours on 8x H100 SXM5, depending on dataset size and sequence length. At the cluster rate of $30.75/hr, a 10-hour run costs $307.50. That is less than the purchase cost of a single used RTX 4090.

For the same run on a 4090 cluster: 70B BF16 requires 8x H100 NVLink for reasonable throughput. An 8x 4090 PCIe cluster can technically run QLoRA on a smaller model but cannot handle 70B BF16 at useful speeds due to PCIe bandwidth. The workloads are not directly comparable.

For inference-only use on 70B models, the H100 SXM5 at $3.84/hr is the single-GPU option. An 8x 4090 cluster at $4.40/hr can technically serve 70B Q4 across cards via tensor or pipeline parallelism (model sharding across cards), but the 32 GB/s PCIe bottleneck makes this impractical for production throughput compared to a single H100's 80 GB on-card memory.

Related Comparisons

If you are deciding between generations of consumer GPUs, see the RTX 5090 vs RTX 4090 AI benchmark for a side-by-side with Blackwell's consumer GPU, including Qwen3 32B throughput and QLoRA fine-tuning up to 30B. If you are on the fence between the H100 and its predecessor, the A100 vs H100 breakdown covers the cost-per-token differences in more detail.

For the RTX 4090 on Spheron, see the RTX 4090 rental page.

Teams that develop on a 4090 and train on H100 on-demand get the best of both worlds: low daily dev costs and full data center capability when they need it. Rent H100 → | Rent RTX 4090 → | View all GPU pricing →

Get started on Spheron →