Cost to Train a 70B Parameter LLM From Scratch (2026)

Training a 70B parameter model from scratch doesn't have a single price tag. It has a range spanning 10 to 20x, and the entire range is explained by one input you control directly: how many tokens you train on relative to the model's size. This post walks through the math that turns a parameter count and a token budget into GPU-hours, then into dollars, using live H200 and B200 cluster pricing on both ends of that range. If you're evaluating whether to fine-tune an existing model instead of pretraining one, see our LLM fine-tuning cost breakdown first, since for most teams that's the right question. This post is specifically about the from-scratch pretraining budget itself, distinct from the broader GPU rent-vs-buy infrastructure decision covered elsewhere on this blog.

The Real Inputs: Token Count, Model Size, and GPU-Hours per FLOP

Every pretraining budget reduces to three numbers: parameter count, token count, and the dollar cost of a GPU-hour. Parameter count is usually fixed by the product goal, you picked 70B for a reason. Token count is the variable that actually decides your budget, and it isn't dictated by any law of nature. It's a design choice with a name.

Chinchilla-Optimal vs Inference-Optimal: Why Llama 3.1 70B Trained on 15T Tokens

Hoffmann et al.'s Chinchilla scaling laws established that for a fixed training compute budget, there's an optimal split between model size and token count that minimizes training loss. That optimal ratio works out to roughly 20 training tokens per parameter. Run the arithmetic on a 70B model and you land at approximately 1.4 trillion tokens as the Chinchilla-optimal token count.

Meta trained Llama 3.1 70B on 15 trillion tokens, over 10x past that number (Llama 3 herd of models paper). This wasn't a mistake or an inefficiency. Chinchilla optimizes training compute for a given loss, and says nothing about inference cost. A model that will be queried billions of times over its deployed lifetime pays an inference cost on every single call that scales with parameter count, while the training cost is a one-time bill. Push a smaller model further past its Chinchilla-optimal token count and you can match a Chinchilla-optimal model's quality at a fraction of the serving cost, for a training budget that's now much larger than "optimal" would suggest. This is the deliberate trade the frontier and open-weight labs have made: overtrain a smaller model because total lifetime cost, not training-run cost, is the thing that actually matters (source).

That's the entire reason this post's cost range spans 10-20x instead of being a single number. The parameter count doesn't move. The token count is a business decision about how much you're willing to spend upfront to save on every inference call for years afterward.

The C = 6ND Formula: Turning Parameters and Tokens Into GPU-Hours

Before you touch a pricing page, you need a FLOPs estimate. The standard approximation for dense transformer pretraining compute is:

C ≈ 6 × N × D

Where C is total training FLOPs, N is parameter count, and D is token count. The factor of 6 accounts for the forward pass (roughly 2 FLOPs per parameter per token) plus the backward pass (roughly 4 FLOPs per parameter per token, since gradients require both an input-gradient and a weight-gradient pass).

For a 70B model:

To convert FLOPs into GPU-hours, divide by a GPU's achieved (not peak) FLOPs/second, then account for the fact that real training runs never hit 100% of peak. H200 SXM5 and B200 SXM6 both publish peak dense BF16/FP16 throughput: H200 at 989 TFLOPS, B200 at 2,250 TFLOPS, a roughly 2.27x per-GPU FLOPs advantage for B200 (source). Worth noting: H200 offers essentially no raw pretraining FLOPs advantage over H100, since they share the same Hopper compute die. H200's real edge is memory bandwidth, 4.8 TB/s versus H100's 3.35 TB/s, which mainly helps larger micro-batches and inference throughput, not the raw FLOPs figure that drives this calculation.

Real clusters achieve 35-45% of peak FLOPs in practice once you account for communication overhead, pipeline bubbles, and checkpointing pauses, what training teams call Model FLOPs Utilization (MFU). Meta's own Llama 3.1 70B run is the empirical anchor here: 15 trillion tokens on a 70B model took approximately 7.0 million H100-80GB GPU-hours (source), out of a 39.3 million GPU-hour total across the full Llama 3 herd (8B at 1.46M, 70B at 7.0M, 405B at 30.84M GPU-hours), run on a cluster of over 16,384 H100 GPUs. That real-world number is what the next section converts into a dollar figure, alongside the Chinchilla-optimal case scaled proportionally from the same FLOPs math.

Reference Cost Range on H200 and B200 Clusters at Current Rates

Here's the same 70B model at both ends of the token-count decision, priced against live GPU cluster rates. All prices below are per-GPU, on-demand, checked against the Spheron GPU pricing API and split strictly by DEDICATED instance type to avoid conflating spot and on-demand rates. GPU-hours are Hopper-class (H100/H200 share the same compute die, so the same figure applies to both); the B200 column applies the same GPU-hour count to B200's higher price as a conservative estimate, since no public MFU-adjusted B200 figure exists yet for a run at this scale, and a real B200 cluster would likely finish in fewer wall-clock hours given its throughput advantage.

The Lean Build: Chinchilla-Optimal 70B at Roughly 1.4T Tokens

Scaling Meta's 7.0M GPU-hour, 15T-token anchor down linearly by token count (since FLOPs scale directly with D for a fixed N) gives roughly 650,000 GPU-hours for a 1.4T-token Chinchilla-optimal run. At the cheapest live H200 SXM5 GPU rental on-demand rate of $5.915/GPU/hr, that's about $3.84 million in raw compute. Applying the same Hopper-class GPU-hour count to B200's higher $7.50/GPU/hr on-demand rate (a conservative estimate, since B200's throughput advantage would likely shrink the real hour count) puts the ceiling near $4.9 million. This is the closest thing to a "budget" 70B pretrain, and it's still a run that needs eight figures of runway once you add data curation, ablations, and staff.

The Overtrained Build: Llama-3.1-Class 70B at 15T Tokens

This is Meta's actual number, not an estimate: 7.0 million H100-80GB GPU-hours for Llama 3.1 70B's 15T-token pretrain (source). Substituting live H200 on-demand pricing of $5.915/GPU/hr gives roughly $41.4 million in raw compute; running the same GPU-hour count on a B200 GPU cluster instead, at $7.50/GPU/hr, pushes the conservative estimate to $52.5 million. Meta didn't rent this cluster hourly, they own it, and building your own 16,384-GPU fleet is a different conversation than this post. But the on-demand-equivalent number is the honest ceiling for what renting the same GPU-hours would cost today, and it's the number that should end most "should we pretrain from scratch" conversations before they start.

For context on the outer bound of frontier-scale training: Stanford's AI Index Report estimated GPT-4's training compute cost at $78 million and Google Gemini Ultra 1.0 at $191 million (source). A 70B-dense pretrain sits well below frontier-model scale, but the overtrained case already clears eight figures.

On-Demand vs Reserved/Committed Cluster Pricing on H200 and B200

Nobody runs a multi-month pretraining job on hourly on-demand pricing if they can avoid it. Reserved and committed cluster capacity carries a meaningful discount over on-demand: roughly 20-30% off for a 6-month commitment, 40-50% off for a 1-year term, and 55-72% off for a 3-year commitment, based on benchmarking across cloud GPU providers (source). Applied to the overtrained 15T-token build, a 1-year reserved H200 rate could plausibly cut that $41.4 million estimate closer to $21-25 million, before any volume negotiation on top of the published term discount.

Pricing fluctuates based on GPU availability. The prices above are based on 25 Jul 2026 and may have changed. Check current GPU pricing → for live rates. If you're at the point of actually negotiating a multi-month reservation, our guide to negotiating GPU cluster reservation contracts covers minimum commitments, burst allowances, and the SLA terms worth pushing back on before you sign.

Where the Budget Actually Goes: Compute vs Data Curation vs Failed Runs

The GPU-hour number above is the floor, not the total. Epoch AI's analysis of frontier training costs breaks down the composition: hardware (chips, servers, interconnect) accounts for 47-67% of total cost, R&D staff 29-49%, and energy just 2-6% (source). That same analysis found the amortized cost of the largest training runs has grown roughly 2.4x per year since 2016, and projects the biggest runs will exceed $1 billion by 2027. A 70B dense model isn't at that frontier scale, but the same cost structure applies proportionally: raw compute is well under the full number you'll actually spend.

Data Curation and Filtering Cost per Trillion Tokens

Raw web-scraped tokens aren't training-ready. Deduplication, quality filtering, and classification pipelines (NeMo Curator, Datatrove, FineWeb-style processing) cost roughly $560-$790 per trillion output tokens on-demand, or $330-$470 per trillion at spot rates, on a 64x H100 cluster (source). For the 15T-token overtrained build, that's roughly $8.4-11.9 million in curation compute alone, on top of the GPU-hour figure above, assuming a curation pass ratio that yields the full 15T tokens as clean output (in practice you curate down from a larger raw pool, so this is a lower bound).

Hardware Failures at Scale: What Meta's 16,384-GPU Run Reveals About Downtime Cost

Interruptions aren't an edge case at this scale, they're the operating condition. During a 54-day snapshot of Meta's Llama 3 405B pretraining on the 16,384-GPU H100 cluster, the team recorded 466 job interruptions: 47 planned and 419 unexpected. Roughly 78% of the unexpected interruptions traced to confirmed or suspected hardware issues, and GPU issues alone caused 58.7% of unexpected interruptions, about one failure every three hours (source). Meta still sustained over 90% effective training time, but only because of automated failure detection and fast checkpoint recovery, engineering investment that isn't in the raw GPU-hour math. A checkpointing strategy that doesn't handle this failure rate turns a three-hour mean-time-between-failures into a real cost line item, not just an annoyance. Teams running on spot or preemptible capacity, where interruptions are the default rather than the exception, need the same discipline; our checkpointing and preemption recovery guide covers the mechanics.

Ablations, Hyperparameter Search, and Why the Final-Run Number Understates the Real Budget

The number everyone quotes, GPU-hours for "the" training run, describes exactly one run: the final one that ships. It doesn't include the smaller-scale ablations run to validate data mixture ratios, learning-rate schedules, or architecture choices before committing the full budget to a single 15T-token attempt. Those ablations typically run at a fraction of full scale, but across enough configurations they add a meaningful multiplier on top of the headline number. Any team budgeting a from-scratch pretrain should plan for this as a distinct line item, not an afterthought absorbed into the final run's cost.

Utilization matters here too. GPU clusters sitting idle between ablation runs, or underused due to scheduling gaps, inflate the effective cost per useful GPU-hour well above the on-demand rate you're paying. The GPU goodput engineering guide covers this from the operator side, including Cast AI's finding that enterprise clusters average just 5% SM utilization when workload scheduling isn't actively managed.

When Continued Pretraining or Fine-Tuning Beats Training From Scratch

For almost every team asking "what would it cost to train our own 70B model," the honest answer is: don't. The gap between a full fine-tune and a from-scratch pretrain isn't a percentage difference, it's several orders of magnitude.

The Cost Gap in One Table

A full fine-tune of Llama 3.1 70B on a 1B-token supervised dataset runs at roughly 8,500-9,400 tokens/sec on a well-tuned 8x H100 SXM5 cluster, finishing in about 30 hours, for a cost of roughly $696 on-demand or around $192 at spot rates (source). That's not a rounding error next to an eight-figure pretrain, it's a different category of decision entirely. If your problem is domain adaptation rather than a fundamentally different base capability, continued pretraining on 50-100B domain tokens sits in between: our continued pretraining guide walks through the B200 VRAM math (roughly 870-890GB unsharded for a 70B model) and multi-node scaling from 8x up to 64x B200.

This isn't a fringe argument. OpenAI itself, in the email announcing the wind-down of its self-serve fine-tuning platform, put it plainly: "Newer base models like GPT-5.5 are much better at following instructions and formats than prior models. Prompt-based approaches are now cheaper and faster, as such, we're seeing fewer use cases that require fine-tuning" (source). If prompting is closing the gap on fine-tuning for many use cases, fine-tuning is closing the gap on from-scratch pretraining even faster: the compute delta between "prompt an existing model" and "fine-tune one" is a few hundred dollars, while the delta between "fine-tune" and "pretrain from scratch" is seven to eight figures.

There's also a documented argument, worth engaging with even though it isn't industry consensus, that most of a from-scratch training budget belongs in data rather than model scale in the first place. One independent ML researcher, citing efficiency gains from FineWeb-Edu, BeyondWeb, and OLMo 3, argues that "efficiency multipliers of 6-9x imply that the majority of compute, approximately 80%, can profitably go to data work" rather than raw parameter or token scaling (source). Whether or not you land on that exact ratio, it's a useful sanity check against any plan that treats "more GPU-hours on the final run" as the only lever available.

A Short Decision Framework for Founders and Infra Leads

Ask these in order before committing to a from-scratch pretrain:

1. Is the problem capability or knowledge? If an existing open-weight 70B model already has the right architecture and general capability, and your gap is domain knowledge or formatting, that's a fine-tuning or continued-pretraining problem, not a pretraining problem. Our VRAM tier guide for self-hosting open-source LLMs is the right starting point for most teams asking this question.
2. Can you afford the ablation budget, not just the final run? If the answer to "what's our budget for data-mixture and hyperparameter ablations before the real run" is "we hadn't planned for that," you're not ready to price a from-scratch pretrain yet, you're underestimating it.
3. Do you have (or can you build) the failure-recovery engineering for a multi-week, multi-node run? Meta's 419 unexpected interruptions in 54 days on a top-tier engineering team's cluster is the baseline, not a worst case. If your team can't build equivalent checkpoint-recovery discipline, a from-scratch run will burn budget on downtime you can't see coming.
4. Have you priced reserved capacity, not just on-demand? The 40-70% swing between on-demand and multi-year committed pricing changes the entire viability conversation for a run measured in months, not hours.

For the overwhelming majority of teams, the honest answer at step 1 ends the conversation. From-scratch pretraining is the right call for a small number of labs with a genuine architectural or capability gap that no existing open model closes, and the budget to match. Everyone else should be evaluating fine-tuning, continued pretraining, or renting inference against an existing model, not pricing out a GPU-hour spreadsheet for a training run they were never going to run.

If the math above points you toward fine-tuning or continued pretraining instead of a from-scratch run, Spheron's H200 and B200 clusters are available on-demand or reserved, with per-minute billing on the on-demand tier.

Check H200 and B200 cluster pricing →