NVIDIA A100 Deployment Guide: SXM vs PCIe, Spot vs Dedicated, and MIG

If you are picking an A100 configuration for a real workload, the hard decisions are not "do I need an A100." They are SXM or PCIe, spot or dedicated, MIG or whole-GPU, and how many cards per node. This guide walks through each of those trade-offs with the numbers and failure modes we see most often in production.

For live A100 pricing and deployment, see the A100 rental page. For a broader architectural comparison, start with our A100 vs V100 analysis or the best NVIDIA GPUs for LLMs framework.

Why the A100 is Still the Default in 2026

The A100 was NVIDIA's first GPU designed specifically for large-scale AI, not just graphics or HPC with AI added on top. Built on the Ampere architecture, it introduced features that reshaped how AI workloads run in production: high-bandwidth HBM2e memory, third-generation Tensor Cores, and Multi-Instance GPU support.

Most importantly, the software ecosystem around A100 is mature. Frameworks like PyTorch, TensorFlow, JAX, TensorRT, and RAPIDS have been tuned for years on this hardware. Engineers know how A100 behaves under sustained load. That reliability still matters more than raw benchmarks.

For many teams, A100 is the most cost-effective way to run serious AI workloads without paying a premium for bleeding-edge hardware they may not fully utilize.

A100 Technical Specifications

The A100 delivers 312 TFLOPS of FP16 Tensor performance, with up to 624 TFLOPS when structural sparsity is enabled. TF32 is particularly important; it allows developers to run FP32 models with much higher performance without rewriting code, delivering major speedups while preserving accuracy. This is one of the reasons A100 saw rapid adoption across existing AI codebases.

Key Architecture Features

Ampere Tensor Cores

At the heart of A100 are NVIDIA's third-generation Tensor Cores, which accelerate matrix operations that dominate AI training and inference. A100 supports multiple numerical formats: FP32, TF32, FP16, BF16, INT8, and INT4, allowing the same GPU to handle training, fine-tuning, and inference efficiently.

HBM2e Memory and Bandwidth

The A100 80GB uses HBM2e memory delivering nearly 2 TB/s of bandwidth. Many AI workloads are memory-bound, not compute-bound; large models, large batch sizes, and long context windows all benefit directly from higher memory capacity and bandwidth. With A100 80GB, more of the model and data stays resident on the GPU instead of spilling into system memory. Our dedicated vs shared GPU memory guide covers what happens the moment a workload exceeds that 80GB and starts paging into system RAM.

NVLink Interconnect

A100 SXM variants support NVLink, NVIDIA's high-speed GPU-to-GPU interconnect. NVLink allows multiple GPUs to communicate at up to 600 GB/s, far faster than PCIe alone. This is critical for multi-GPU training, model parallelism, and large inference clusters. When GPUs exchange gradients or activations quickly; scaling efficiency improves and training time drops.

Multi-Instance GPU (MIG)

MIG allows a single A100 GPU to be split into up to seven isolated GPU instances, each with its own memory, compute, and bandwidth allocation. These instances behave like independent GPUs. This is extremely useful for inference workloads, shared environments, and teams running multiple smaller jobs; instead of underutilizing a full GPU, MIG lets teams pack workloads efficiently while maintaining isolation.

Model Capacity on A100

A100 80GB handles models up to roughly 30B parameters in FP16 for training (with optimizer states). For inference, quantized 70B models fit on a single A100 80GB. The sweet spot is 7B to 30B parameter models where the A100 provides ample memory without requiring multi-GPU sharding. Check our best NVIDIA GPUs for LLMs guide to see if A100 matches your specific model requirements.

40GB vs 80GB: The VRAM Decision

The 40GB vs 80GB choice is about whether your model and its runtime overhead fit comfortably, not just technically.

Bandwidth gap. A100 SXM4 80GB delivers 2,039 GB/s of memory bandwidth. A100 SXM4 40GB delivers 1,555 GB/s - 24 percent less. For autoregressive decoding, which is almost always memory-bandwidth-bound, the 80GB variant generates tokens faster even when the model fits in 40GB. NVIDIA's benchmarks show up to 1.25x higher inference throughput on the 80GB vs the 40GB for batch-size-constrained models like RNN-T automatic speech recognition running inside MIG partitions; compute-bound workloads see a smaller or no gap.

Where 40GB runs out. A 30B model in BF16 needs 60GB for weights alone - that exceeds the 40GB limit entirely. A 13B model in BF16 needs 26GB, leaving 14GB for KV cache and activations - workable for short contexts but tight at production batch sizes. The 40GB variant is genuinely useful for 7B to 13B inference with context windows under 8K tokens.

KV cache at long context. At 32K context, a 70B INT4 inference request generates approximately 10GB of KV cache. Added to the 35GB model footprint, that totals 45GB - already over the 40GB card's capacity. At 64K context, KV cache grows to approximately 21.5GB, pushing the combined total to 56.5GB. The 80GB variant handles both context lengths comfortably; the 40GB cannot sustain a 70B INT4 request at either once KV cache overhead is included.

Two 80GB GPUs for 70B. Two A100 80GB GPUs connected over NVLink provide 160GB combined - the minimum for BF16 70B inference with realistic context lengths and the starting point for LoRA fine-tuning of a 70B model at BF16 precision. If 70B models are on your roadmap, size for two 80GB cards with NVLink from the start. Migrating a running inference service when you hit a memory wall mid-production is painful.

When 40GB still makes sense. If your workload is exclusively 7B to 13B inference with short contexts and you never need multi-GPU tensor parallelism, the 40GB variant covers you and is often cheaper per hour. Outside that narrow band, the 80GB is the safer default.

Choosing an A100 Configuration

A100 comes in four configurations you will actually encounter on GPU clouds: 40GB or 80GB memory, SXM4 or PCIe form factor. The right pick depends on workload shape and blast radius.

A100 80GB SXM4 is the default for multi-GPU training, large-batch fine-tuning, and anything that needs NVLink. Picks up to 600 GB/s between GPUs, 400W per card, 2.0 TB/s memory bandwidth. This is what ships in DGX A100 and HGX A100 reference designs.

A100 80GB PCIe is the right call for single-GPU inference, data processing, or mixing GPUs into standard servers without the thermal and power budget of SXM. 300W, no NVLink, same compute and memory subsystem.

A100 40GB (SXM or PCIe) still exists but is rarely the right pick in 2026. The 80GB variant has the same compute with 2x the memory and ~30 percent more bandwidth. The only reason to choose 40GB is if it is meaningfully cheaper per hour on a workload that genuinely fits.

On Spheron, the A100 80GB runs at $1.43/hr on-demand (dedicated) and $1.19/hr on spot. As a frozen snapshot from 6 Sep 2026, AWS p4de A100 instances ran $3.43 per hour, Azure ND A100 v4 ran $4.10 per hour, and Google Cloud A100 ran $5.07 per hour - all more than double the live Spheron rate above that day. Those hyperscaler figures are not live-checked, so treat the comparison as directional; see the live Spheron rate above or the pricing page for current numbers.

Pricing fluctuates based on GPU availability. Spheron rates above are live as of 17 Oct 2026; other providers reflect their most recent published rates and may have changed. Check current GPU pricing -> for live rates.

For current per-hour pricing across these configurations, see the A100 rental page.

Understanding SXM vs PCIe

This distinction matters more than most providers explain.

Bandwidth. A100 SXM4 80GB delivers 2,039 GB/s of memory bandwidth. A100 PCIe 80GB delivers 1,935 GB/s - about 5 percent less. For most single-GPU workloads, that gap does not move the needle. For memory-bandwidth-bound jobs like large-batch autoregressive generation, it shows up in tokens per second.

NVLink vs PCIe interconnect. This is where the real difference lives, though it is more nuanced than most comparisons suggest. SXM4 cards connect over NVLink at 600 GB/s total bidirectional bandwidth via NVSwitch. PCIe A100 cards support an NVLink Bridge accessory that also reaches 600 GB/s between two GPUs - matching SXM4 at that scale. The distinction is how far each scales: SXM4 with NVSwitch maintains full NVLink bandwidth across 4 to 8 GPUs in a node, while PCIe NVLink Bridge tops out at 2 GPUs. Beyond 2 cards, the PCIe path falls back to PCIe Gen4 at 64 GB/s for inter-GPU traffic.

Power. SXM4 draws 400W per card. PCIe 80GB draws 300W. A 4-card PCIe node pulls 400W less at peak load. If you are fitting GPUs into standard server chassis with standard power distribution, PCIe unlocks configurations that SXM physically cannot use.

Decision matrix:

If you are running a 70B model across 2 A100 PCIe cards with NVLink Bridge, you get the same 600 GB/s GPU-to-GPU bandwidth as SXM4 at that node size. For 4-GPU or 8-GPU clusters, SXM4 is the only path to full NVLink bandwidth across all cards. A PCIe-only multi-GPU node can also hit a scaling ceiling well short of what the GPU count promises once GPUs split across separate NUMA domains, as this dual-socket 8-GPU NUMA postmortem found. If you are running a 13B model serving API calls on one GPU, PCIe saves 100W per card with no throughput penalty worth measuring.

MIG in Production

MIG (Multi-Instance GPU) turns one physical A100 into up to 7 isolated compute instances. Each instance gets dedicated SM compute slices, L2 cache, memory, and memory bandwidth - hardware-enforced, not just OS-level isolation. Faults in one partition do not affect the others.

Valid profiles on A100 80GB:

On the A100 40GB, the same slice fractions apply but memory scales proportionally: the smallest profile is 1g.5gb (5GB per instance, 7 instances max) and the full-GPU profile is 7g.40gb. You cannot mix profile sizes on one GPU unless the combination sums to exactly 7 SM slices with compatible memory slice counts.

Multi-tenant inference. Seven 1g.10gb instances on one A100 80GB each handle a 7B model in INT4 with memory headroom to spare. That is 7 independent inference processes - one per customer, one per service, one per team - with hardware isolation from a single physical card. A memory error or crash in one instance does not touch the others.

When MIG makes sense:

• Packing multiple 3B to 7B inference services onto one card with isolation guarantees
• Multi-tenant SaaS where customer workload isolation is a contractual or compliance requirement
• Development environments where multiple engineers share one production-grade GPU
• Cost optimization when your serving workload only needs 10 to 20GB, not 80GB

When MIG does not help. MIG instances cannot use NVLink for cross-GPU communication. If your model needs tensor parallelism across cards, MIG is the wrong tool entirely. Some workloads - certain video pipelines and CUDA libraries that assume full-GPU access - do not run inside MIG partitions. Verify your stack's MIG compatibility before building production infrastructure around it.

Enabling MIG on Linux (NVIDIA driver 470+):

# Enable MIG mode on GPU 0 (requires stopping GPU processes)
sudo nvidia-smi -i 0 -mig 1

# List available profiles
sudo nvidia-smi mig -lgip

# Create 7 instances of 1g.10gb (use profile name; numeric IDs change across driver versions)
sudo nvidia-smi mig -cgi 1g.10gb,1g.10gb,1g.10gb,1g.10gb,1g.10gb,1g.10gb,1g.10gb -C

# List created instances and their UUIDs
sudo nvidia-smi mig -lgi

Assign instances to containers via CUDA_VISIBLE_DEVICES=MIG-<uuid>. Each container sees only its assigned partition. For vLLM or TGI serving, spin a separate process per MIG instance with the appropriate environment variable set.

Spot vs Dedicated: Picking the Right Tier

On Spheron, on-demand A100 comes in two tiers: dedicated (99.99% SLA, non-interruptible) and spot (cheaper, interruptible). Spot is cheaper because it gets reclaimed when demand spikes. Dedicated is more expensive because it does not. The choice is workload-shaped.

Use spot for: training jobs with frequent checkpointing (every 15 to 30 minutes), batch inference, data preprocessing, hyperparameter sweeps, anything you can safely resume. The cost savings are large enough that even losing an hour of progress once a day usually still comes out ahead.

Use dedicated for: production serving, jobs where re-running costs more than the instance, anything with a deadline. The predictability is worth the premium.

A practical pattern that works well: run most of your training on spot, keep a small dedicated tier as a failover for the last 10 to 20 percent of any long run where rerun cost would exceed the spot savings.

Common Use Cases

Training and fine-tuning: A100 handles models in the 7B to 30B parameter range comfortably. It supports mixed-precision training (TF32, FP16, BF16) and FSDP for larger models. Teams use it for continued pre-training, supervised fine-tuning, and LoRA/QLoRA.

Production inference: A100 delivers stable latency and high throughput for production serving. MIG allows teams to isolate workloads cleanly, which improves utilization and reduces operational complexity. If inference cost-per-token is your primary metric, see NVIDIA L40S vs A100 for a direct FP8 vs BF16 throughput comparison.

Data analytics: RAPIDS, GPU-accelerated SQL engines, and cuDF benefit directly from A100's memory bandwidth and CUDA ecosystem. Teams running data preprocessing pipelines alongside training see significant speedups.

Research and experimentation: Startups and researchers use spot A100 instances to experiment quickly without committing to expensive long-term infrastructure. The mature ecosystem means most papers and codebases "just work" on A100. That "just works" claim mostly holds for PyTorch specifically. If your team is still deciding which framework to standardize on for A100 workloads, the PyTorch vs TensorFlow comparison covers where each one's CUDA support and ecosystem tooling actually differ in practice.

When A100 Is the Right Choice

A100 makes sense when you want reliable AI compute at a reasonable cost. It fits teams that value stability, ecosystem maturity, and predictable performance. Power draw is lower than H100-class GPUs (400W vs 700W), which keeps total cost of ownership closer to flat. See our GPU cost optimization playbook for practical tactics.

If your workloads are hitting memory bandwidth limits or need models beyond 70B in FP16, H100 or H200 may be a better fit. Otherwise, A100 remains one of the smartest choices in the GPU market. Compare options in our GPU renting on Spheron and the best NVIDIA GPUs for LLMs framework.

Ready to deploy? Check live per-hour A100 pricing and configurations on the A100 rental page or see all GPU pricing.

A100 GPU on Spheron →