GPU Hours Fine-Tuning Comparison: LoRA vs Full Fine-Tune on a 70B Model (2026)

Every LoRA vs full fine-tuning comparison you'll find stops at a VRAM table or a single dollar figure per run. Neither one tells you the number that actually decides whether a 70B fine-tuning job is worth scheduling: GPU-hours. A GPU-hours fine-tuning comparison has to track two things that pull in different directions, how much compute each method burns per training step and how many GPUs each method forces you to rent just to fit the model in memory. LoRA cuts the first number by roughly a third. It does not cut the second number by a third, not even close, and closing that gap is the point of this post. For the VRAM math this post builds on, start with GPU VRAM requirements to fine-tune LLMs if you haven't sized your model and method yet.

TL;DR: GPU Hours Fine-Tuning Comparison for LoRA vs Full Fine-Tuning on a 70B Model

• GPU-hours: Full fine-tuning a 70B model on 11x H100 SXM5 takes about 352 GPU-hours; LoRA on 2x H100 SXM5 takes 16-28, roughly a 16x gap.
• GPU count: LoRA fits on 2 H100s; full fine-tuning needs 11, since Adam's optimizer states alone need about 560 GB of VRAM at 70B.
• FLOPs: LoRA uses just over two-thirds the FLOPs per pass (Thinking Machines Lab), so the 16x gap is about GPU count, not compute.
• Verdict: LoRA is the cheap default for most tasks; full fine-tuning's extra GPU-hours pay off on code- and math-heavy jobs where LoRA's accuracy risk is highest.
• Spheron: rents H100 SXM5 at $2.98/hr on-demand as of 05 Nov 2026, scaling 1-11 GPUs, no contract. See the fine-tuning guide.

GPU-Hours Fine-Tuning Comparison: LoRA vs Full Fine-Tune, Measured

GPU-hours, not wall-clock hours, is the unit that maps straight to your bill, because you're paying for every GPU in the cluster for every hour it's reserved, whether or not that particular GPU is the bottleneck at any given moment. Spheron's own fine-tuning cost breakdown, recorded on 7 Jul 2026 and frozen so the arithmetic holds, gives three real data points for a 70B model:

Full fine-tuning's wall-clock time is only about 2.9x longer than LoRA's midpoint, 32 hours against roughly 11. The GPU-hours gap is far larger, about 16x, because full fine-tuning also ties up 5.5x as many GPUs for that whole stretch. Multiply a modest wall-clock penalty by a large GPU-count penalty and you get the real number a budget has to absorb.

Why LoRA Isn't Proportionally Faster Per Step

If GPU-hours tracked FLOPs, this would be a much smaller story. In its LoRA Without Regret research note, Thinking Machines Lab put a precise number on the per-step compute difference: "LoRA takes slightly more than ⅔ of the FLOPs that full fine-tuning does per pass." Invert that and full fine-tuning should cost somewhere around 1.5x LoRA's compute per step, not 16x its GPU-hours.

The disconnect is that FLOPs per step and GPU-hours per job are answering different questions. FLOPs tell you how much arithmetic a GPU performs; GPU-hours tell you how many GPUs you had to rent and for how long. LoRA's FLOP savings come from only computing gradients and running backward passes through a small set of adapter matrices. Full fine-tuning's GPU-hours bill comes from a completely separate constraint: it needs to store gradients and Adam optimizer states for all 70 billion parameters, and that storage requirement, not the step's arithmetic, is what forces an 11-GPU cluster instead of a 2-GPU one. A method can use less compute per step and still need vastly more total GPU-hours, because the thing rationing your GPU count is memory capacity, not FLOPs.

70B Wall-Clock: 2x H100 SXM5 LoRA vs 11x H100 SXM5 Full Fine-Tune

The GPU count split, 2 versus 11, comes straight from VRAM, not from a vendor recommendation. The VRAM sizing tables linked above put 70B LoRA's footprint at roughly 159 GB (BF16 weights plus adapter gradients and optimizer states), which clears a single H100's 80 GB but not by enough to risk it, so 2x H100 SXM5 (160 GB combined) is the practical minimum. Full fine-tuning's footprint at the same model size is roughly 860 GB, which needs 11x H100 SXM5 (880 GB combined) before FSDP or ZeRO-3 sharding has anywhere to put the model at all.

That 2-versus-11 split is also why the wall-clock gap, 2.9x, is so much smaller than the GPU-hours gap, 16x. Wall-clock time answers "how long until the job finishes." GPU-hours answers "how much capacity did I reserve to get there." Full fine-tuning finishes in not much more than 2.9x the time, but it does so by occupying 5.5x the hardware the whole way through, and that second multiplier is the one that shows up on the invoice.

Where Full Fine-Tuning's VRAM Overhead Actually Costs You

The GPU-count gap traces back to one component more than any other: the optimizer state.

The Optimizer-State Bill: 560 GB of Adam Moments at 70B

Adam and AdamW store two FP32 running statistics per trainable parameter, a first moment and a second moment, at trainable_params × 4 bytes × 2 moments regardless of your training dtype. At 70B, full fine-tuning trains all 70 billion parameters, which works out to roughly 560 GB just for those optimizer moments. LoRA and QLoRA train only their adapter parameters, typically under 1% of the base model, which keeps optimizer state down to roughly 5.6 GB for the same 70B base. That single component is most of the difference between a cluster that needs 2 GPUs and one that needs 11.

It's also why switching from LoRA to QLoRA doesn't shrink optimizer memory further: both train the same small set of adapter parameters, so both carry the same roughly 5.6 GB optimizer-state bill at 70B. QLoRA's advantage is in the frozen base model weights, quantized to 4-bit instead of kept in BF16, not in the optimizer.

FSDP Sharding Across 11 GPUs Adds Communication Time LoRA Never Pays

Spreading 860 GB across 11 GPUs doesn't just divide the memory, it adds work that a 2-GPU LoRA job never has to do. FSDP and DeepSpeed ZeRO-3 shard parameters, gradients, and optimizer states across every GPU in the cluster, which means every forward and backward pass has to all-gather the full parameter shards from every other GPU before it can compute, then reduce-scatter the gradients back out afterward. That round trip runs on every layer boundary, on every step, for the entire 32-hour job, and its cost scales with how many GPUs are in the cluster and how fast they can talk to each other over NVLink or InfiniBand, not with how much useful compute any one step is doing.

LoRA avoids almost all of this. Its adapter gradients are a few hundred million parameters instead of 70 billion, small enough to synchronize across 2 GPUs with ordinary data-parallel all-reduce, no full-parameter reconstruction required on every layer. The FSDP, DeepSpeed ZeRO-3, and Megatron-Core setup guide covers the sharding mechanics and NCCL tuning behind that overhead in more depth if you're configuring the cluster yourself; the short version for this post is that full fine-tuning's GPU-hours bill is paying for communication time LoRA structurally never incurs, on top of the extra GPUs.

The Accuracy Tradeoff: When LoRA's Cheaper Run Isn't Worth It

None of the GPU-hours math above answers whether the cheaper run gets you a model worth deploying. That's a separate question, and the honest answer at 70B is that the published research doesn't fully answer it either.

Code and Math Are Where the Gap Shows Up Most

Databricks and MosaicML's LoRA Learns Less and Forgets Less study is the clearest evidence on this. It fine-tuned models on programming and mathematics tasks, under both an instruction-tuning regime of roughly 100K examples and a continued-pretraining regime of 20 billion tokens, and found that in standard low-rank settings, the ones most teams actually run, LoRA substantially underperforms full fine-tuning's target-domain accuracy, with a bigger gap on code than on math. The same study found LoRA forgets less of the base model's source-domain performance than full fine-tuning does, in both task types, so the tradeoff cuts both ways: LoRA buys resistance to forgetting at the cost of some target-domain accuracy, and that cost lands harder on code than on math.

None of those runs were at 70B specifically, the models tested were smaller, but code and math are exactly the task types where the research flags a real gap. That's the scale where the GPU-hours savings calculated earlier in this post are measured fact while the accuracy parity is still an open question your own eval set needs to answer before you bank on it.

None of this means LoRA can't hit production-quality accuracy. It means the method's tuning and the task matter more than the method's name. Spheron's case study on 10 concurrent fine-tuning jobs ran QLoRA, not full fine-tuning, on Llama 3.3 70B across 10 enterprise customers and moved customer-task accuracy from 67.9-76.3% up to 87.3-93.8%, with MMLU moving only -0.6 to +0.1 points. That's a parameter-efficient method closing a large accuracy gap when it's matched correctly to a task that isn't code generation or formal math, which is a useful counterweight to the code and math findings above: the accuracy risk is concentrated on specific task types, not spread evenly across every LoRA use case.

DoRA and Other PEFT Variants as a Middle Ground

If plain LoRA's accuracy risk on your task is real but an 11-GPU full fine-tune is overkill, there's a middle tier worth checking before you commit to either extreme. Spheron's own DoRA, GaLore, PiSSA, and VeRA decision guide tested DoRA (Weight-Decomposed Low-Rank Adaptation), which splits the weight update into a direction component, handled by a LoRA-style low-rank matrix, and a magnitude component, a single learnable scalar per output dimension, and found it typically closes 1-3% of accuracy against plain LoRA for about 5-10% more VRAM. The same guide found GaLore goes further: it performs true full-parameter gradient updates by projecting gradients into a low-rank subspace before computing the optimizer state, cutting Adam's memory cost by 65-75% versus full fine-tuning while using roughly 30-40% more VRAM than LoRA, and it can match or beat full fine-tuning quality on a range of benchmarks. Neither needs anywhere near an 11-GPU cluster. That guide has the full VRAM tables and recipes for all four if one of them fits your accuracy budget better than plain LoRA.

Renting the Right GPU Count for Either Path

Once you know whether you're renting 1, 2, or 11 GPUs, the GPU generation you rent them in moves the GPU-hours number more than most teams expect, because LoRA and QLoRA runs at this scale are VRAM-bound more than compute-bound.

Framework choice moves the GPU-hours number too, on either method: Axolotl, Unsloth, and TorchTune show real throughput differences on identical hardware, and that's a lever worth pulling before you pay for a bigger GPU count. On the full fine-tuning side specifically, H100's throughput gain over A100 is worth its higher hourly rate on a multi-GPU cluster more often than it is for a 2-GPU LoRA job, since full fine-tuning's FSDP communication overhead scales with GPU count the way the earlier section describes.

Spheron rents both ends of this split on the same marketplace: a single H100 for QLoRA, 2x H100 SXM5 for LoRA, or an 11-GPU H100 SXM5 cluster for full fine-tuning, billed per minute after a short minimum with no long-term contract, at live rates aggregated across 5+ providers. Spot capacity is shown as an explicit discount off on-demand rather than blended into one number, which matters more on the 11-GPU full fine-tuning job than on a single-GPU LoRA run: losing one node mid-FSDP-shard loses the whole job unless checkpointing is enabled, since spot capacity can be reclaimed without notice. The marketplace doesn't publish a fine-tuning-specific SLA or a pre-tuned multi-node training image either, so you're still assembling your own FSDP, Axolotl, or Unsloth stack on top of the rented capacity. On-demand and spot rates move independently, so check the live rate for your configuration before budgeting either path.

A Worked GPU-Hours Budget for Both Methods

Putting the GPU-hours and cost numbers from this post side by side, using Spheron's recorded 7 Jul 2026 rates for the historical reference point and live tokens for today's rate:

Pricing fluctuates based on GPU availability. Spheron rates above are live as of 05 Nov 2026; the 7 Jul 2026 reference costs are historical and frozen so the arithmetic holds. Check current GPU pricing → for live rates.

Read the GPU-hours column against the accuracy section above before picking a row. QLoRA and LoRA are the cheap, fast path, and they're the right call for most instruction-tuning and domain-adaptation jobs that aren't code generation or formal math at the limits of what the base model can represent. Full fine-tuning's 352 GPU-hours is the number to accept, not avoid, when your task is exactly the kind the research above flags as LoRA's weak spot, or when you've already run a LoRA pass and it isn't closing the gap your eval set demands.

Whichever row you land on, the GPU count is the real decision, not just the hourly rate next to it. Check H100 availability →