verl's own quickstart trains PPO on a single GPU with 24GB of HBM. Running verl GRPO from chat agent instructions instead of a terminal session changes almost everything except that first step: swap the algorithm to GRPO, point the reward at a coding environment where a real unit test suite grades each rollout instead of a learned reward model, and wire the whole job, GPU, config and reward path, through one chat prompt to an MCP-connected agent that provisions its own GPU, launches the run, and reports the reward curve back.
TL;DR: Verl GRPO From Chat Agent on a Coding RL Environment
Running verl GRPO from chat agent prompts means an MCP-connected agent provisions a GPU, launches verl with the Sandbox Fusion reward path pointed at a unit test suite, and relays the reward curve and a reward-hacking check back to chat.
- Algorithm: verl's GRPO docs show it trains no critic: it samples multiple rollouts per prompt (
rollout.n>1), normalizes reward within each group, and folds KL into the loss (kl_loss_coefdefaults to 0.001) instead of subtracting it from the reward. - Reward path: a coding environment grades rollouts through verl's Sandbox Fusion endpoint (
sandbox_fusion.url, ending in/run_code) and theprimereward manager, scoring real unit test pass or fail. - GPU floor: verl's documented quickstart runs PPO on one GPU with 24GB+ HBM; a GRPO coding run sampling several rollouts per prompt needs more than that floor implies.
- Spheron: the agent runs the whole job through Spheron's Agent GPU Sandbox on an H100 on-demand instance or larger, with a price preview before every deploy.
The Prompt: Asking the Agent to Train a Coding Policy With Verl GRPO Graded by Unit Tests
Neither Spheron's Agent GPU Sandbox feature page nor its docs publish a worked prompt for this exact job, so here's one built from verl's own GRPO and Sandbox Fusion flags rather than lifted from either:
Deploy a GPU VM, clone volcengine/verl, and stand up a Sandbox Fusion server for code execution. Configure a GRPO run on a small open coding model with no critic,
rollout.n=8group samples per prompt, and the reward routed throughreward_model.sandbox_fusion.urlwith theprimereward manager, so each rollout's generated patch runs against the task's unit tests. Start it as a background job. Tell me which GPU you picked, the reward curve every 20 steps, and flag anything that looks like the policy is gaming the grader instead of actually fixing the code.
An agent reading that pulls the base model and dataset, stands up a Sandbox Fusion server, and translates "no critic, 8 samples per prompt" into the real GRPO flags before the first rollout fires. None of that data moves through the chat window: Spheron's sandbox caps chat-to-VM file transfer at 5MB per file, so a coding dataset of real size comes from git or huggingface-cli on the VM, the same way it would from a terminal.
Which GPU the Agent Picks, and Why a Coding Environment Needs the Sandbox Fusion Reward Path
A GRPO coding run needs more GPU than verl's own quickstart suggests, because that 24GB floor is sized for PPO on a half-billion-parameter model doing GSM8K math, not a coding policy sampling several rollouts per prompt against a real test suite. verl's GRPO config reuses the same actor training path and memory math, minus the critic plus group sampling, and each extra rollout in the group is another sequence's worth of activations and KV cache held in memory at once. The honest floor for a coding run is "more than 24GB," not 24GB. Given the sandbox's catalog, H100 through B300 plus A100, L40S and RTX cards, an H100 instance is the realistic starting point for a small coding model before reaching for a B200 or B300.
The reward path is where a coding environment differs from a math or logic one. verl's Sandbox Fusion integration routes each rollout's code to an execution endpoint, reward_model.sandbox_fusion.url, which has to end in /run_code, with a max_concurrent limit. Pairing that with reward_model.reward_manager=prime verifies code across subprocesses concurrently, the same category of reward as a verifier-based environment: a binary or scalar signal from a deterministic checker, not a trained scorer.
One thing a single sandbox VM doesn't give you is the CPU/GPU split most larger agentic RL systems use. Spheron's agentic RL infrastructure post covers why most teams run the rollout fleet on CPU sandboxes and keep the GPU for the trainer; one VM folds both onto a single node by design, fine at quickstart scale, wrong once rollouts need to scale independently of the trainer.
Watching GRPO Train: Pass Rate, Reward Curve, and the Reward-Hacking Checks Logged to Chat
A background job on the sandbox runs up to 24 hours and streams output as it goes. For a multi-hour run, the agent has to summarize rather than dump, which is what the prompt above asked for: a reward curve every 20 steps, not a line-by-line transcript. Worth watching: the group-normalized reward trending up, the unit test pass rate climbing with it, and the KL term (folded into the loss rather than the reward, per verl's GRPO config) staying bounded instead of drifting, usually the first sign the policy is straying from the reference model.
The harder thing to catch is a policy climbing the reward curve without getting better at the task. The practical version for a chat-run job: have the agent check whether a passing patch is lifting the fix from history rather than reasoning to it, and don't rely on held-out tests alone.
Is It Safe to Let an Agent Run Unit Tests With Sudo on a Rented GPU?
The sudo access here is for setup, cloning the repo, installing the Sandbox Fusion server, pulling the dataset, not for the unit tests themselves. verl's Sandbox Fusion path already runs each rollout's code inside its own execution instance with a memory_limit_mb cap and a concurrency limit, the isolation boundary doing the real work during training, not the VM's root access. Spheron's sandbox adds a second layer of control around the agent driving it: every deploy, Stop, Start and terminate needs your explicit approval, a connection can carry its own spend limit, the VM opens no inbound port, and every command is logged with its text and exit code. Your own SSH key keeps working, so you can tail the real log yourself instead of trusting only the agent's summary. Spheron's CUDA kernel debugging walkthrough covers these same controls in more depth, worth reading once rather than re-explained here.
The genuine limit: it's one VM per connection, not a cluster, so a run needing a trainer and several rollout engines across GPUs at once is bounded by whatever single node you deploy, and the 5MB chat-transfer cap means a real dataset still comes from git or huggingface-cli on the VM, never through chat.
What It Costs to Run Verl GRPO From Chat Agent Against a Coding Environment
The job costs the GPU's normal hourly rate for however long the background job runs, with no separate sandbox fee. An H100 on-demand instance on Spheron runs $3.60/hr as of 20 Nov 2026, so a multi-hour GRPO check on a small coding model is a few hours of that rate, and a bigger model or deeper rollout group scales with however many hours and however large a GPU the agent picked.
Two things to plan around before starting the clock. Checkpoint the run: on a different training job on Spheron, not a GRPO one, an 8x H100 spot node with no volume and no checkpoints was stopped at step 956, 17 minutes in, and the $10.22 it had already cost bought nothing back, because that provider's spot nodes can't have a volume added after launch. A second node with a 100GB network volume checkpointing every 500 steps limited a later reclaim to at most 500 steps lost, the number to aim for on a run you can't afford to lose hours of rollouts from. And prices move while you watch them. Check the live listing at deploy time, not from whatever number you saw yesterday.
Pricing fluctuates based on GPU availability. Spheron rates above are live as of 20 Nov 2026; other providers reflect their most recent published rates and may have changed. Check current GPU pricing → for live rates.
Where This Fits Next: SkyRL, SWE-Gym, and Learned Reward Models
A single sandbox VM running verl GRPO against one unit-test-graded environment is the right scale for a quick check of a coding policy, not for training a frontier coding agent on a dataset the size of SWE-Gym. Spheron's SkyRL on GPU cloud guide covers the real GPU counts SkyRL's launch scripts need past the single-GPU sandbox check.
Two other paths branch off here. If the task doesn't reduce to a pass or fail unit test, no verifiable reward exists to route through Sandbox Fusion, and you need a learned reward model instead; Spheron's reward model training walkthrough is the same chat-to-sandbox pattern on that problem. Optimizing a full multi-step tool-use trajectory rather than a single patch is covered in Spheron's agent RFT guide, with the trajectory-level reward design a single-shot coding environment doesn't need.
Verl's GRPO config and Sandbox Fusion reward path are already built for this; what you need next is somewhere to run them without babysitting a terminal for the next several hours.
Frequently Asked Questions
GRPO (Group Relative Policy Optimization) trains no critic model. [verl's GRPO docs](https://verl.readthedocs.io/en/latest/algo/grpo.html) set it with algorithm.adv_estimator=grpo and actor_rollout_ref.rollout.n>1, sampling several rollouts per prompt and normalizing the reward within that group instead of learning a separate value function. It also folds the KL term into the loss (use_kl_loss=True, with kl_loss_coef defaulting to 0.001) rather than subtracting a KL penalty from the reward the way some PPO setups do. PPO, by contrast, trains an actor and a critic together and estimates advantage from the critic's value predictions.
A GRPO run against a coding environment, sampling several rollouts per prompt through a real unit test suite, needs real headroom.
[verl's Sandbox Fusion integration](https://verl.readthedocs.io/en/latest/examples/sandbox_fusion_example.html) routes each rollout's generated code to an execution endpoint (reward_model.sandbox_fusion.url, which must end in /run_code) with a max_concurrent limit. Pairing this with reward_model.reward_manager=prime verifies code across multiple subprocesses concurrently.
It's a policy getting credit for a fix it didn't really produce.







