Training a reward model used to mean standing up a multi-GPU cluster before you'd written a line of RewardTrainer code: provision the boxes, install the CUDA stack, pull the preference dataset, and only then discover your batch size was wrong. A GPU sandbox reward model run skips most of that. Spheron's Agent GPU Sandbox lets an agent provision its own GPU from a single chat prompt, run TRL's RewardTrainer itself, and report the loss curve back to you. This walkthrough covers the exact prompt, which GPU the agent picks for a LoRA-adapted pairwise reward model, the metrics it streams back mid-run, the dollar cost, and where the resulting reward model plugs into a GRPO or PPO pipeline next.
TL;DR: GPU Sandbox Reward Model Training
A GPU sandbox reward model run trains a LoRA-adapted pairwise reward model on one GPU, provisioned and run entirely from a chat prompt.
- Mechanism: Spheron's Agent GPU Sandbox connects at
connect.spheron.ai/mcp, provisions one GPU VM, and runs TRL's RewardTrainer as a background job. - GPU: an 8B-scale LoRA reward head fits a single H100 on-demand, far short of the 8-GPU node a full 70B reward model needs.
- Loss: RewardTrainer optimizes the Bradley-Terry loss, logging loss, accuracy, margin, and mean reward every step.
- Controls: every deploy and terminate needs approval, and the VM opens no inbound port.
- Cost: Spheron H100 on-demand runs $3.96/hr as of 08 Nov 2026; a single-GPU LoRA run like this finishes in a few hours.
The Prompt: What You Type to Train a Reward Model From Chat
The workflow starts the same way the LoRA fine-tuning walkthrough does: add the Agent GPU Sandbox as an MCP server once, then send the instruction in chat.
claude mcp add --transport http spheron https://connect.spheron.ai/mcpSpheron's own Agent GPU Sandbox feature page documents fine-tuning prompts like "Fine-tune Qwen3 8B on my support tickets and tell me the eval loss," with GPU tiers from H100 up to B300, plus A100, L40S, and RTX cards. Neither that page nor its docs carry a dedicated example for reward-model training, so the prompt below is composed from that same documented pattern paired with TRL's own RewardTrainer API, not quoted from Spheron's materials:
Train a reward model on Qwen3 8B using TRL's RewardTrainer. My preference
data is at ~/datasets/support-preferences.jsonl with chosen and rejected
fields, about 8,000 pairs. Use LoRA with modules_to_save for the reward
head, train for 2 epochs on an H100, and report the accuracy, margin, and
final loss.Nothing in that message specifies a batch size or a learning rate schedule. The agent fills in RewardConfig's own defaults, batch size 8 per device, 3 epochs unless capped lower, a 1e-4 learning rate for a full fine-tune, gradient checkpointing and bf16 on by default, unless you override them. The dataset itself has to be pulled directly on the VM: file transfer between chat and the sandbox is capped at 5MB per file, so anything past a toy-sized preference set comes down by git clone, wget, or huggingface-cli download, the same way TRL's own quickstart pulls the public trl-lib/ultrafeedback_binarized preference set for its documented example run.
Which GPU the Agent Picks for a Pairwise Reward Model Run
The agent doesn't guess at a GPU tier from nothing. It matches your prompt against the sandbox's supported hardware, H100, H200, B200, B300, plus A100, L40S, and RTX cards, and provisions whichever tier the instruction names, or a safe default when you leave it unspecified. For an 8B base model trained as a reward model, that default is the same H100 the sandbox's documented fine-tuning prompts reach for, and for a LoRA-adapted run it's real headroom rather than the minimum. It's also worth being clear about what the sandbox isn't: a GPU VM here takes a minute or two to boot after you approve the deploy preview, fine for a multi-hour training job and the wrong tool if you just need a fast, sub-second CPU-only grading environment.
Why LoRA on the Reward Head Changes the GPU Math
RewardTrainer loads the base model through AutoModelForSequenceClassification and forces num_labels to 1, so what gets trained is structurally a classifier with one scalar output, not a generation head. Training that classifier end to end, full-parameter, means storing BF16 weights, BF16 gradients, and FP32 Adam optimizer states for every parameter in the base model. Spheron's own RLHF infrastructure guide puts the recommended setup for a 70B reward model at 8x H100 SXM5 or 8x B200, because the frozen, inference-only shape the reward model takes later, during PPO, isn't the shape it trains in. Training it is a standard fine-tune with a full optimizer footprint, and a 70B reward model on that setup typically takes 18-30 hours depending on dataset size and sequence length.
LoRA changes what actually needs gradients. [TRL supports training reward models with PEFT adapters directly, and recommends setting modules_to_save=['score']](https://huggingface.co/docs/trl/en/reward_trainer) in the LoraConfig so the reward head trains alongside the adapter rather than staying frozen with the rest of the base model; adapter-only runs typically use a higher learning rate, around 1e-3, than the 1e-4 default RewardConfig ships for a full fine-tune. At 7B scale with LoRA at rank 64 on a BF16 base, that's roughly 20GB against roughly 88GB for a full fine-tune of the same model, per Spheron's GPU VRAM requirements guide, which fits a single H100 with room to spare. That gap is the entire reason a reward-head LoRA job stays a single-GPU sandbox session instead of the multi-node setup the 70B full-parameter case needs.
Watching the Run: Loss, Accuracy, and Checkpoints Reported in Chat
Once training starts, the agent runs it as a background command, not a blocking one. The sandbox's default command timeout is 10 minutes, nowhere near enough for even a short RewardTrainer run, so a reward-model job has to be an explicit background call with output streaming back to chat. A single background command can run for up to 24 hours before it needs restarting, comfortable headroom for a single-GPU LoRA reward-model run at 8B scale.
What streams back resembles the step logs any RewardTrainer run prints to its own console: loss, accuracy (how often the chosen response scored higher than the rejected one), min_reward, mean_reward, max_reward, margin, learning_rate, and grad_norm, logged at whatever interval your training arguments set. Because the Bradley-Terry loss is underdetermined, adding a constant to every reward doesn't change which response wins a pairwise comparison, RewardConfig exposes a center_rewards_coefficient term that pulls mean rewards toward zero, recommended at 0.01. The agent relays the latest lines rather than the full stream: only the first 1MB of any command's output returns to chat, with the complete log saved to `/var/log/spheron-agent/` on the VM. Checkpoints land on the same disk at whatever save interval you set, and because the base model and any adapters persist on the VM between tasks, you can ask the agent to resume from the last checkpoint and report its saved path back into chat in a follow-up message instead of re-downloading an 8B model from scratch.
Is It Safe to Train on Preference Data With Sudo Access?
The honest answer mirrors every other job this sandbox runs: it's built to bound the risk, not remove it. The agent runs commands as the login user with sudo, which is what lets it install TRL, transformers, and PEFT, and pull a preference dataset with git or huggingface-cli, without you opening a terminal yourself first. Several controls sit around that access: every deploy, Stop, Start, and terminate shows a price preview and needs your explicit approval, only the person who deployed the instance can use its sandbox, the VM opens no inbound port, and a connection can carry a spend limit per hour so a misconfigured training loop can't run up an open-ended bill. Every command the agent runs is logged with its text and exit code, and your own SSH key keeps working throughout, so you can tail the actual RewardTrainer log yourself mid-run instead of trusting only the agent's chat summary. None of that changes based on what the data contains; the same approval gates, sudo scope, and spend limit apply whether the job is training on preference pairs or debugging a failing CUDA kernel on the same sandbox. More detail on each control is in Spheron's Agent GPU Sandbox docs.
What It Costs to Train a Reward Model From Chat
The sandbox itself carries no separate fee. What you pay for is the GPU instance's normal hourly rate for however long it stays running:
- Instance rate: Spheron H100 on-demand runs $3.96/hr as of 08 Nov 2026, the same rate you'd pay renting that GPU directly.
- Run time: a single-GPU LoRA reward-model run at 8B scale sits well inside the sandbox's 24-hour background ceiling; a dataset in the low thousands of preference pairs typically finishes in a few hours rather than needing the full day.
- Idle time: the meter keeps running between the job finishing and you approving Stop or terminate, so the real bill is training time plus however long the chat sits unattended afterward.
- The contrast case: a full-parameter 70B reward model on 8x H100 SXM5 runs 18-30 hours and multiplies both the GPU count and the hourly rate, which is the gap a LoRA-adapted reward head exists to close for anything short of frontier scale.
Pricing fluctuates based on GPU availability. Spheron's H100 rate above is live as of 08 Nov 2026; the 70B reward-model runtime figure reflects benchmarks recorded on their own stated date and may have changed. Check current GPU pricing → for live rates.
What a GPU Sandbox Reward Model Feeds Into: GRPO and RLHF Pipelines
The reward model this run just produced isn't the end of the pipeline, it's an input to the next one. A frozen version of it becomes the scorer in a full PPO loop, running alongside an actor, a critic, and a reference model, four roles with four different memory shapes running simultaneously. Spheron's RLHF infrastructure guide covers that sizing in detail: a 70B actor and critic each need full training stacks with optimizer states, while the frozen reference and reward models run inference-only. That's a heavier, multi-node job the sandbox described here doesn't orchestrate end to end; it provisions one VM per connection, and a four-role PPO loop needs several.
Before reaching for that pipeline at all, it's worth checking whether the task needs a learned reward model in the first place. GRPO eliminates the critic and the separate reward model entirely for tasks with a verifiable reward, a unit test passing, a math answer matching, a format matching a regex, by scoring rollouts directly against that verifier instead of a trained scorer. If your reward signal can be written as a deterministic function, GRPO is almost always the cheaper path. A reward model, LoRA-trained from a GPU sandbox or otherwise, earns its cost specifically when the thing you're scoring, tone, helpfulness, which of two answers reads better, can't be reduced to a pass or fail check.
The reward model in this walkthrough trained on a single GPU because an agent could provision one, run TRL's RewardTrainer, and hand the checkpoint back without anyone opening a terminal.
Frequently Asked Questions
A reward model is a scalar scoring function trained to predict which of two responses a human or AI judge preferred, so it can stand in for that judgment during reinforcement learning. TRL's RewardTrainer trains it as an Outcome-supervised Reward Model: it loads the base model with AutoModelForSequenceClassification, forces num_labels to 1, and backpropagates the Bradley-Terry negative log-likelihood, -log(sigmoid(reward(chosen) - reward(rejected))), on a dataset of chosen and rejected response pairs.
Yes, if you adapt the base model with LoRA instead of a full fine-tune. TRL supports training reward models with PEFT adapters directly and recommends setting modules_to_save=['score'] so the reward head trains alongside the adapter, with a higher learning rate around 1e-3 for adapter-only runs. An 8B-scale LoRA reward head fits comfortably on a single H100; a full-parameter reward model at 70B needs the base weights, gradients, and Adam optimizer states for every parameter, which is why Spheron's RLHF infrastructure guide recommends 8x H100 SXM5 or 8x B200 for that scale instead.
The sandbox is built to bound the risk, not remove it. Every deploy, Stop, Start, and terminate shows a price preview and needs your explicit approval, only the person who deployed the instance can use its sandbox, the VM opens no inbound port, and a connection can carry a spend limit per hour. Every command the agent runs is logged with its text and exit code, and your own SSH key keeps working throughout, so you can tail the training log yourself mid-run instead of trusting only the agent's summary.
Reward model training is a standard fine-tune: one forward pass, one backward pass, a gradient update, on preference pairs. PPO is a different workload entirely, running an actor, a critic, a frozen reference model, and the now-frozen reward model simultaneously to generate rollouts, score them, and update the policy. GRPO sidesteps this by eliminating the learned reward model for tasks with a verifiable reward, scoring rollouts against a deterministic checker instead of a trained scorer.
