Tutorial

Claude Code Multi-GPU Fine-Tuning: Full Fine-Tune From Chat (2026)

Back to BlogWritten by Published Nov 19, 2026
Claude Code Multi-GPU Fine-TuningFull Fine-Tune From ChatAgent GPU Sandbox Multi-GPU TrainingMulti-GPU Full Fine-TuningFSDP Multi-GPU Fine-Tuning24-Hour Background Job LimitAI Agents
Claude Code Multi-GPU Fine-Tuning: Full Fine-Tune From Chat (2026)

Ask Claude Code to full fine-tune a 70B model and two limits collide before the first checkpoint saves. One is memory: full fine-tuning stores gradients and optimizer states for every parameter, forcing the agent onto a real multi-GPU node, not the single card a LoRA job gets away with. The other is time: that run's own wall-clock length can outlast a single background command inside the sandbox. Claude Code multi-GPU fine-tuning from chat means navigating both at once, and this walkthrough covers the mechanism end to end: the prompt, the node the agent picks, the framework it wires in by hand, and what happens when the job outruns the clock.

TL;DR: What Full Fine-Tuning From Chat Looks Like

  • VRAM: Full fine-tuning a 70B model needs ~860 GB of VRAM, versus ~159 GB for LoRA and ~52 GB for QLoRA, forcing a multi-GPU node.
  • The sandbox ceiling: Spheron's Agent GPU Sandbox caps a background command at 24 hours, 6-8 hours short of that 30-32 hour run.
  • Spheron: An 11x H100 SXM5 cluster runs $39.60/hr on-demand as of 20 Nov 2026. Check H100 availability.

Why Claude Code Multi-GPU Fine-Tuning Needs Multiple GPUs (and Why Unsloth Can't Do It Alone)

Full fine-tuning updates every parameter in the model, so it has to store BF16 weights, BF16 gradients, and two FP32 Adam optimizer moments for all of them. At 70B, that totals roughly 860 GB: 140 GB weights, 140 GB gradients, and about 560 GB of optimizer state. LoRA and QLoRA only train a small set of adapter parameters, which is why their footprint at the same model size lands at roughly 159 GB and 52 GB, both of which fit a single card. Full fine-tuning doesn't; the math alone is what Spheron's VRAM requirements guide walks through across model sizes. Even an 8B model isn't exempt: full fine-tuning an 8B model needs roughly 88 GB of VRAM, which clears a single H100's 80 GB only by splitting the model across 2 GPUs with sharding.

That memory wall is exactly where Unsloth stops being an option on its own. Unsloth's own documentation is direct about it: multi-GPU setups currently work "through libraries like Accelerate and DeepSpeed," and official multi-GPU support is still something the team says "we'll be announcing... soon." An agent asked to full fine-tune a model past single-GPU scale has to reach for Accelerate, DeepSpeed, or PyTorch's own FSDP, covered in Spheron's FSDP, DeepSpeed ZeRO-3, and Megatron-Core guide, and configure one of them itself. No flag turns a single-GPU Unsloth script into a sharded multi-node job.

The Prompt and Which Multi-GPU Node the Agent Picks

The workflow starts the same way the LoRA version does, one message after the sandbox is connected:

bash
claude mcp add --transport http spheron https://connect.spheron.ai/mcp

A prompt that actually asks for a full fine-tune has to say so explicitly, because full fine-tuning and LoRA produce completely different GPU requests:

text
Full fine-tune Llama 3.3 70B on my dataset at ~/datasets/support-corpus.jsonl.
Use FSDP across an H100 SXM5 cluster sized for a full fine-tune, not LoRA.
Checkpoint every 500 steps and tell me the loss and run time when it finishes.

GPUs on the Agent GPU Sandbox run from RTX cards and A100 up to H100, H200, B200, and B300, and are available as single or multi-GPU configurations on most offers. For a 70B full fine-tune specifically, the agent needs a cluster at or above the 11x H100 SXM5 minimum this post's TL;DR gives, since anything smaller can't hold 860 GB even before FSDP has room to shard activations. The sandbox's own documentation doesn't name specific multi-GPU node sizes or confirm true multi-node orchestration across separate machines; what it describes is multiple GPUs inside one provisioned VM, so the actual GPU count on offer has to be checked at deploy time. Naming the method and cluster size in the prompt matters: the agent deploys what you ask for rather than silently downgrading a full fine-tune request to the single-GPU, LoRA-sized node the sandbox's LoRA walkthrough covers. A true multi-node launch, as opposed to multiple GPUs inside one VM, still needs MASTER_ADDR and MASTER_PORT set by hand, since SLURM does not set MASTER_ADDR automatically, so the agent has to script that part itself.

Watching a Run That Pushes the 24-Hour Background Job Limit

Once FSDP or DeepSpeed ZeRO-3 is wired in and training starts, the agent runs it as a background command, the same way it would for a LoRA job. The difference is duration. A 70B full fine-tune on 11x H100 SXM5 runs about 32 hours wall-clock, roughly 352 GPU-hours, against a single background command's ceiling of 24 hours before it needs restarting. That's not a close call; it's a 6-8 hour overrun on a job that has to run uninterrupted to reach a usable checkpoint.

In practice this forces a checkpoint-and-resume pattern across more than one background command rather than a single call. The agent launches training with checkpointing enabled (every 500 steps is a reasonable interval), lets the background command run until it approaches the 24-hour mark, and the next chat turn tells it to resume from the last checkpoint instead of restarting from step zero. Because packages, the downloaded base model, and checkpoints persist on the VM between tasks, resuming doesn't mean redownloading a 70B model, just picking the training loop back up.

Is It Safe to Hand an Agent Sudo on a Multi-GPU Node?

The risk profile here is the same one the LoRA walkthrough covers, just on a bigger node. Every deploy, Stop, Start, and terminate shows a price preview and needs explicit approval first, the VM opens no inbound port and only talks outward over HTTPS, and only the account that deployed the instance can use its sandbox. A per-connection spend limit bounds what an unattended multi-hour job can run up, which matters more here than on a single-GPU LoRA run, since an 11-GPU cluster bills for all 11 GPUs every hour it's up, whether training is progressing or stalled. The one real constraint worth flagging: file transfer between chat and the VM is capped at 5 MB per file, so a multi-node launch script or a dataset of any real size has to be pulled directly on the machine, by wget or a storage client the agent runs itself, not pasted through chat. Your own SSH key keeps working throughout, so you can tail the FSDP log yourself instead of relying only on the agent's summary. More detail on each control is in Spheron's Agent GPU Sandbox docs.

What Claude Code Multi-GPU Fine-Tuning Costs

The sandbox itself carries no separate fee; you pay the cluster's normal hourly rate for however long it stays up. An 11x H100 SXM5 cluster, the minimum for a 70B full fine-tune, runs $39.60/hr on-demand as of 20 Nov 2026 ($3.60/hr per GPU). A 32-hour run at that rate is the number to budget against, not the per-GPU figure alone, since all 11 GPUs bill for the full duration regardless of which one is the bottleneck. Spheron's LoRA vs full fine-tuning GPU-hours comparison puts that run at roughly 352 GPU-hours against 16-28 for a 70B LoRA job on 2 GPUs, close to a 16x gap, which is the clearest way to see why a prompt that says "full fine-tune" asks for a materially different bill than one that says "LoRA."

Pricing fluctuates based on GPU availability. Spheron rates above are live as of 20 Nov 2026; other figures are from benchmarks recorded on their own stated dates and may have changed. Check current GPU pricing → for live rates.

Getting an agent onto a genuine multi-GPU node, and letting it manage a run that outlasts a single background command, only works if the sandbox itself can provision real multi-GPU hardware and keep state between chat turns.

Connect your agent to the Agent GPU Sandbox →

FAQ / 04

Frequently Asked Questions

Not on its own. Claude Code runs on CPU; what gives it GPU access is connecting to a provider's GPU VM over MCP. Through Spheron's Agent GPU Sandbox at connect.spheron.ai/mcp, the agent provisions a GPU VM itself, from RTX cards and A100 up to H100, H200, B200, and B300, and runs sudo commands on it for as long as the task needs.

Yes, when the node it provisions has more than one. Spheron's Agent GPU Sandbox offers single or multi-GPU configurations on most offers, so a prompt naming a multi-GPU tier gets the agent a node where it still has to install and configure a sharding framework (FSDP, DeepSpeed ZeRO-3, or Accelerate) itself, since a multi-GPU node and a working multi-GPU training job are two different problems.

Unsloth's own documentation says multi-GPU setups currently work only through external libraries like Accelerate, DeepSpeed, FSDP, or DDP, and that official multi-GPU support is still coming. An agent asked to full fine-tune a 70B model has to wire one of those frameworks in manually; Unsloth alone only ever uses one GPU.

A single background command in the sandbox runs for up to 24 hours before it needs restarting. A 70B full fine-tune's documented ~30-32 hour wall-clock time structurally cannot finish in one call, so the agent has to checkpoint, let the command end, and resume from that checkpoint in a follow-up prompt rather than running the whole job as a single background task.

// Try It Yourself

Try It on Real GPUs

The GPUs behind these guides are the ones you can rent here: H100s, H200s, B200s, and more, billed per minute after a 20-minute minimum runtime, with no contracts. Pick one and you are live in under two minutes.

View Pricing
→
Or talk to our team
Spheron GPU instance going live and linking data centers across the globe