L40S vs H100 for AI Inference: When the L40S Wins on Cost Per Token (2026 Decision Guide)

Choosing between L40S and H100 for inference is not as simple as comparing price-per-hour. L40S at $0.72/hr on Spheron is 5.33x cheaper than H100 SXM5 at $3.84/hr, and H100 delivers 2-3x more tokens per second at batch 1 and 3-5x more at batch 32+. But even at batch 32, H100's throughput advantage does not close the 5.33x price gap, so L40S wins on cost-per-token in most serving scenarios. H100 wins for 70B+ model inference on a single card, NVLink-required training, and long-context workloads.

This post works through the specs, FP8 throughput benchmarks for Llama 3.1 8B and Qwen3 14B, the cost-per-million-token math using live Spheron pricing, and a concrete decision framework for which GPU belongs in your stack. For operators comparing L40S against A100 instead, see the L40S vs A100 cost-per-token guide which runs the same analysis against the Ampere architecture.

TL;DR: When L40S Beats H100

The honest summary: L40S wins on cost-per-token in most LLM serving scenarios because the 5.33x price difference exceeds H100's 2.7x-5x throughput advantage. H100 wins for 70B+ model single-card inference, NVLink-required training, and long-context workloads.

Specs: L40S vs H100 Side by Side

The bandwidth difference is the number that matters most for inference. At 864 GB/s (L40S) vs 3,350 GB/s (H100 SXM5), H100 moves model weights and KV cache 3.9x faster. This is the primary throughput driver at low batch sizes.

Architecture: Ada Lovelace vs Hopper

FP8: Same Feature, Different Ceiling

Both L40S and H100 support FP8 via NVIDIA's Transformer Engine. The engine dynamically selects FP8 or BF16 precision per layer at runtime, using per-tensor scaling to maintain accuracy. For practical inference with vLLM, both GPUs support the same --quantization fp8 flag and produce equivalent output quality.

The gap is throughput capacity. H100 SXM5 delivers 3,958 FP8 TFLOPS with sparsity. L40S delivers 1,466. That is 2.7x more FP8 compute per GPU. At batch sizes where the workload is compute-bound (roughly batch 16+ for 7B-13B models), this advantage directly translates into more tokens per second.

For a deep dive into H100's FP8 Tensor Core architecture, the H100 complete datasheet covers all precision tiers and the Transformer Engine implementation.

Memory Subsystem: GDDR6 vs HBM3

H100 SXM5 uses HBM3 (High Bandwidth Memory, 3rd generation) at 3,350 GB/s. L40S uses GDDR6 at 864 GB/s. The technology difference matters for two reasons.

First, bandwidth. HBM stacks DRAM dies directly on the GPU package, enabling the wide memory interface that delivers 3.35 TB/s. GDDR6 uses conventional PCB-mounted DRAM with a standard memory bus, topping out at 864 GB/s on L40S.

Second, capacity. HBM's die-stacking architecture enables larger capacities in the GPU thermal envelope. H100 SXM5 packs 80GB of HBM3 on the package. L40S carries 48GB of GDDR6. For 70B parameter models, this 32GB gap is the decisive factor: 70B FP8 (70GB) fits on a single H100 but not on a single L40S.

At low batch sizes (batch 1-4), inference is memory-bandwidth-bound. GPU utilization is low; the bottleneck is how fast weights and KV cache can be loaded into SRAM for each forward pass. H100's 3.9x bandwidth advantage directly caps how much L40S can close the throughput gap here.

NVLink: Only H100 Has It

H100 SXM5 supports NVLink 4.0 at 900 GB/s total bidirectional bandwidth per GPU via NVSwitch. L40S is PCIe-only. For multi-GPU inference where tensors need to be exchanged between cards (tensor parallelism), NVLink makes a real difference. PCIe 4.0 provides roughly 64 GB/s bidirectional on a 16-lane slot, about 14x less bandwidth than NVLink.

For tensor-parallel inference on 70B models (where you're splitting the model across 2+ GPUs), H100's NVLink dramatically reduces communication overhead. For training runs with all-reduce gradients, NVLink is close to required for efficient multi-GPU scaling. The L40S's PCIe limitation is not a problem for single-card inference or embedding serving, but it is a meaningful constraint for multi-GPU training.

Benchmark: Cost Per Token at Batch 1, 8, and 32

The throughput figures below are representative estimates derived from community vLLM benchmarks and extrapolated from the L40S FP16 baseline (46 tok/s at batch 1, 336 tok/s at batch 8, Llama 3.1 8B) with FP8 approximately doubling throughput at memory-bound regimes and providing additional gains at compute-bound regimes. Results will vary with vLLM version, prompt length distribution, and system configuration. Use these as directional guidance, not authoritative benchmarks.

Cost formula: cost_per_million = (hourly_rate / tok_per_second) * (1_000_000 / 3600)

Llama 3.1 8B FP8

At batch 8, L40S at ~$0.31/M tokens is 2.16x more cost-efficient than H100 at ~$0.67/M tokens, despite H100's higher throughput. At batch 32, L40S at ~$0.13/M tokens still beats H100 at ~$0.24/M tokens. H100's 3x throughput advantage at batch 32 does not overcome the 5.33x price difference, so L40S wins on cost-per-token across all batch sizes measured here.

Qwen3 14B FP8

For 14B models, the bandwidth bottleneck is more pronounced at batch 1 and 8 because larger weights require more memory bandwidth per forward pass. L40S VRAM is sufficient (14B FP8 = ~14GB of weights, leaving 34GB for KV cache), and while the bandwidth gap means H100 delivers about 3.3x more throughput at batch 8, L40S at ~$0.69/M tokens still beats H100 at ~$1.25/M tokens because the 5.33x price difference is larger than the throughput gap.

The cost-per-token calculation here uses live GPU pricing from Spheron as of the date below. Check the pricing page for current rates before making infrastructure decisions.

Pricing fluctuates based on GPU availability. The prices above are based on 26 May 2026 and may have changed. Check current GPU pricing → for live rates.

When L40S Wins

Low-Utilization Endpoints (7B-30B Models)

The strongest case for L40S is when your inference endpoint runs at moderate, bursty, or unpredictable traffic. You pay per GPU-hour, not per token. An H100 sitting idle at $3.84/hr is more expensive than an L40S sitting idle at $0.72/hr. Even at full utilization, L40S wins on cost-per-token for 7B-30B models at typical batch sizes, so lower utilization only widens the advantage further.

For a concrete example: an internal demo endpoint serving 30 requests/hour at average 200 tokens/response is generating 6,000 output tokens/hour. Both GPUs handle this trivially; neither is anywhere near saturation. Monthly GPU cost: L40S at $0.72 720 hours = $518.40. H100 at $3.84 720 hours = $2,764.80. L40S saves $2,246.40/month.

For a detailed guide on vLLM deployment on L40S, including configuration flags and container setup, see NVIDIA L40S for AI inference.

Image Generation (SDXL, Flux)

SDXL and Flux.1 diffusion workloads are compute-bound on the UNet/DiT backbone, not memory-bandwidth-bound in the same way as autoregressive LLMs. Both L40S and H100 complete a single SDXL inference pass in broadly similar wall-clock time, with H100 having some advantage from its higher compute density.

The practical difference is small enough that for single-tenant image generation endpoints, L40S generates images at a comparable rate to H100 at 5.33x lower cost per hour. The 48GB VRAM is sufficient for SDXL at any standard resolution and for Flux.1 up to 1080p. For parallel batch image generation at high concurrency, H100's compute advantage does scale up, but most image generation deployments are latency-focused rather than batch-throughput-focused.

Embedding Serving

Embedding models (BGE, GTE, sentence-transformers) are small relative to the L40S's 48GB capacity and computationally light. A single L40S can serve millions of embeddings per hour with very low GPU utilization. Running embedding endpoints on H100 is functional but wastes most of the GPU's capabilities.

For multi-modal embedding pipelines or retrieval-augmented generation (RAG) preprocessing, L40S handles the embedding workload at $0.72/hr vs $3.84/hr for an H100. The throughput per dollar for embeddings strongly favors L40S because neither GPU is the bottleneck; the CPU-side tokenization and batching typically are.

Multi-Tenant SaaS with Per-GPU Isolation

If you're building a SaaS product that gives tenants dedicated GPU access rather than shared inference, cost per GPU-hour is your primary metric. L40S at $0.72/hr lets you offer per-GPU dedicated instances at lower price points than H100.

For 7B-14B models, one L40S per tenant is sufficient for typical serving workloads. MIG partitioning is not available on L40S, so you cannot subdivide it. But for small models, you don't need MIG; the full 48GB card handles a 14B FP8 model with plenty of KV cache headroom.

When H100 Wins

70B+ Model Inference

70B FP8 weights total 70GB. H100 SXM5's 80GB HBM3 fits the model on a single card with 10GB remaining for KV cache at short context lengths. L40S at 48GB requires two GPUs (96GB total) for 70B FP8.

For 70B+ models, a single H100 at $3.84/hr avoids the PCIe tensor-parallel overhead that 2x L40S ($1.44/hr combined) incurs. While 2x L40S is cheaper on paper at $1.44/hr, the PCIe-limited communication between cards significantly cuts throughput for large model inference, making H100's single-card performance and NVLink scaling the better choice for 70B+ workloads.

Large Batch Serving: H100 Throughput Without Cost-Per-Token Win

At batch sizes of 32 and above, the workload transitions from memory-bandwidth-bound to compute-bound. H100's 3,958 FP8 TFLOPS (2.7x the L40S's 1,466) pull ahead in raw throughput, delivering roughly 3x more tokens per second at batch 32 for 7B-13B models.

With H100 at $3.84/hr and L40S at $0.72/hr, H100 would need 5.33x the throughput of L40S to break even on cost-per-token. A 3x throughput advantage does not clear that threshold. As the benchmark tables show, L40S wins on cost-per-million-tokens at batch 32 even for compute-bound workloads. For teams where throughput ceiling matters more than cost-per-token (hard latency SLOs, burst handling), H100's raw speed may still justify the higher rate.

Training and Fine-Tuning

NVLink is the deciding factor for multi-GPU training. H100 SXM5's NVLink at 900 GB/s total bidirectional bandwidth enables efficient all-reduce gradient synchronization across a training cluster. L40S's PCIe-only connectivity caps inter-GPU bandwidth at roughly 64 GB/s, making multi-GPU gradient synchronization roughly 14x slower. For LoRA fine-tuning of 13B-70B models on 2-8 GPU setups, H100 multi-GPU training converges significantly faster than L40S multi-GPU training.

For single-GPU fine-tuning of small models (7B with LoRA or QLoRA), L40S is perfectly capable. But anything requiring multi-GPU gradient synchronization belongs on H100.

Long-Context Inference (32K+ Tokens)

KV cache size scales linearly with sequence length, layer count, and batch size. For a 13B model at 32K context with batch 8, KV cache alone can consume 20-30GB. Combined with model weights (13GB FP8), this pushes against L40S's 48GB ceiling and requires aggressive chunked-prefill or reduced batch size to prevent OOM.

H100's 80GB HBM3 handles large KV caches comfortably, and its 3.35 TB/s bandwidth processes the attention mechanism significantly faster at long contexts. For RAG pipelines with long retrieved contexts or conversational endpoints with extended history, H100 is the safer and faster choice. For long-context inference optimization see the KV cache optimization guide.

Spheron Pricing: L40S vs H100

Live prices from Spheron as of 26 May 2026:

For comparison, hyperscaler H100 rates:

Spheron aggregates compute from 5+ providers into a single marketplace, which is how it offers rates below hyperscaler prices. Both GPUs are available on-demand with per-minute billing and SSH root access.

For batch inference workloads under 30B parameters at moderate utilization, L40S GPU rental at $0.72/hr provides capable FP8 inference at the lowest available price point. Teams running 70B+ inference or multi-GPU training workloads should rent H100 on Spheron where on-demand single-GPU availability makes it straightforward to right-size the allocation.

Pricing fluctuates based on GPU availability. The prices above are based on 26 May 2026 and may have changed. Check current GPU pricing → for live rates.

Migration Patterns

Start on L40S, Add H100 When Needed

The practical approach for teams building inference infrastructure: start on L40S for development and lower-traffic serving, then add H100 nodes when throughput demands grow. L40S and H100 instances on Spheron are provisioned independently, so you can run a mixed fleet without migrating the entire deployment.

A common pattern is routing by model size at the load balancer: small and mid-size models (7B-13B) stay on L40S, while 70B+ requests route to H100. This keeps cost-per-hour low for the majority of traffic while handling large model requests on the appropriate hardware.

The L40S vs A100 comparison has a step-by-step migration playbook that also applies when moving from L40S to H100.

Mixed Fleet Strategy

For teams serving multiple model sizes, a mixed fleet lets you right-size per workload:

• L40S nodes: embedding models, SDXL/Flux image generation, 7B-13B LLM serving at moderate traffic
• H100 nodes: 70B+ model inference, high-throughput batch serving, fine-tuning jobs

Route at the load balancer by model ID and estimated batch size. For most teams, this hybrid approach delivers lower total infrastructure cost than running everything on H100 while still having H100 capacity available for workloads that justify it.

Decision Checklist

Before choosing, work through these questions:

• Model size: is your primary model 70B+ parameters? If yes, H100 on a single card avoids PCIe tensor-parallel overhead that 2x L40S incurs.
• Batch size: what is your P50 batch size? Below 16 means you're memory-bandwidth-bound, where the L40S/H100 gap is proportional to bandwidth ratio (3.9x). Above 32 means compute-bound, where the gap is 2.7x TFLOPS. Even so, L40S wins on cost-per-token at both regimes due to the 5.33x price difference.
• Context length: are you serving prompts longer than 16K tokens at batch 8+? If yes, H100's 80GB handles KV cache headroom L40S lacks.
• GPU utilization: what is your average GPU utilization over a 24-hour window? Below 40-50% means L40S saves heavily on absolute spend, but even at full utilization L40S wins on cost-per-token for 7B-30B models.
• Training vs inference: is this a serving workload only? For fine-tuning on multi-GPU setups, H100's NVLink is decisive.
• NVLink requirement: do you need tensor-parallel or pipeline-parallel training across multiple GPUs? L40S has no NVLink.
• VRAM headroom: does your model plus KV cache at target context length and batch size fit in 48GB? Use the formula in howToStep 3 to verify.
• Spot tolerance: can your workload tolerate interruptions? Both H100 SXM5 (spot $1.69/hr) and L40S (spot $1.07/hr) have spot pricing on Spheron. Note that L40S spot is currently priced above L40S on-demand ($0.72/hr), which is unusual and likely reflects temporary supply conditions. For H100, spot at $1.69/hr is significantly below on-demand at $3.84/hr, making H100 spot an attractive option for interruption-tolerant workloads.

If your inference workload runs 7B-30B models at moderate traffic where GPU utilization stays below 50%, L40S on Spheron cuts your hourly GPU spend by 81% compared to H100 SXM5 on-demand. For high-utilization batch serving or 70B+ models, H100's throughput and single-card capacity win out. Check live availability and pricing for both.

Rent L40S → | Rent H100 → | View all GPU pricing →