Eval Infra & Sandboxing
Reward verification is the hard part. The right eval strategy shapes your cost and latency.
RLVR stands for Reinforcement Learning with Verifiable Rewards. The "verifiable" part is load-bearing: instead of a neural reward model, you have a deterministic checker — the math answer is correct or it isn't, the code passes the tests or it doesn't.
What the checker looks like — and what it costs — depends entirely on the task.
Reward cost by task type
The eval cost spectrum spans six orders of magnitude — from a 1ms regex to a 10-hour GPU training run. Where your task sits on this spectrum determines whether eval is free or the dominant cost.
A sympy or regex checker running on the CPU. Essentially free — never the bottleneck. This is the happy path for RLVR cost.
Model-generated code runs in a sandbox against unit tests. Light tests (competitive programming, LeetCode) finish in milliseconds. Heavy test suites with I/O and subprocesses can take 5–30s — longer than the rollout generation itself.
KernelBench tasks require compiling and profiling CUDA kernels — you need a GPU in the sandbox, not just a CPU. Each eval compiles the kernel, checks correctness on 5 random inputs, then profiles 100 iterations. Individual kernels run in 1–100ms, but compilation + profiling takes tens of seconds. Kevin-32B (Cognition) trained with 16 parallel trajectories and 4 refinement turns — eval stalling accounted for ~55% of total compute time.
TerminalBench and SWE-bench tasks give the agent a bash shell inside a Docker container. Each trajectory is multi-turn (up to 50–100 steps, 32K–64K tokens). A lightweight SWE-bench run averages ~3.5 min per instance; thorough runs can exceed 2 hours. DeepSWE spawned 512 Docker containers per RL iteration across 1,000+ CPU cores, with a 20-min timeout per generation. RollArt (Alibaba) measured ~6 min per iteration, with environment init alone consuming 15% of wall-clock (and up to 78% on failures).
The most ambitious RLVR tasks require actually running training or optimization as the reward signal. NanoGPT speedrun evaluates each agent attempt by training GPT-2 on 8xH100 — the current record is ~3 min, but unoptimized code takes 45 min. Each RL rollout burns real GPU-hours on training. PostTrainBench goes further: each trajectory is a full post-training pipeline (data prep, SFT, sometimes GRPO) with a 10-hour budget on a single H100. At these timescales, the eval is the cost — rollout generation is a rounding error.
# Eval time per rollout (8B model, H100)
──────────────────────────────────────────────────────
Math grader (sympy): ~1ms ✓ free
Code sandbox (warm, light tests): ~5-10ms ✓ fine
CUDA kernel (compile + profile): ~seconds ⚠ needs GPU
SWE-bench (lightweight agent): ~3.5 min ⚠ container
SWE-bench (thorough agent): ~10-120m ✗ dominates
NanoGPT speedrun (8xH100): ~3-45 min ✗ GPU-hours
PostTrainBench (full pipeline): ~2-10 hr ✗ eval IS cost
──────────────────────────────────────────────────────
As eval time grows, async dispatch and off-policy tolerance become mandatory
Sandbox backends
When you need to execute model-generated code, you need a sandbox. The three main options sit on a spectrum from free-but-you-manage to managed-but-you-pay.
| Backend | How it works | Warm latency | Cost |
|---|---|---|---|
| SandboxFusion | Local Docker container you run yourself. No external API calls. Used in veRL's official examples. | ~10ms | $0 (your CPU) |
| Modal (CPU) | Cloud microVMs with gVisor isolation. Pool of warm sandboxes amortizes cold starts. Good for code-RL and SWE tasks. | ~5ms | ~$0.10–0.20/hr per sandbox |
| Modal (GPU) | Same microVM isolation but with GPU attached. Needed for KernelBench-style CUDA compilation, NanoGPT eval, or any task where the reward requires running on a GPU. | ~5ms | ~$2.80/hr (A100-40GB) |
| Daytona | Managed sandboxes with snapshot/fork/resume. ~90ms creation. Can stop and archive state between runs. | ~90ms | ~$0.07/hr per sandbox |
SandboxFusion is the simplest choice for code-RL: spin up a Docker container, point your reward function at localhost:8080, done. Modal CPU is the go-to when you don't want to manage infra — for SWE tasks and competitive programming. Modal GPU is the step up for tasks like KernelBench where the reward itself needs a GPU: at ~$2.80/hr per A100 sandbox, running 32 parallel GPU sandboxes for a 10-hour KernelBench training run adds ~$900 in eval cost alone. Daytona is newer to the RL space but its snapshot/fork model is a natural fit for agentic tasks where each trajectory needs persistent state — they have a TRL+GRPO integration guide.
The cost formula is simple: sandbox-seconds per eval × rollouts per step × $/sandbox-second. For math tasks this is $0. For code-RL with light tests, sandbox cost is a rounding error. For agentic tasks with heavy test suites running hundreds of containers in parallel, it can rival the GPU bill.
Other reward signals
Not every task has a binary verifier. Several other reward types are common in RLVR — and they're all cheap enough that they rarely move the cost needle.
A regex or parser check: did the model put its answer inside \boxed{}, use the right XML tags, follow the required JSON schema? Pure string matching — zero cost, zero latency. Often combined with a correctness reward (e.g. 0.1 for correct format + 0.9 for correct answer).
For writing quality, instruction following, helpfulness — tasks without a binary answer. A capable LLM scores each rollout. The cost lever is judge model size: a small fine-tuned judge (3–8B) often achieves 90%+ agreement with GPT-4o on well-defined rubrics at ~30x lower cost.
# Cost per step (G=16, 512-token completions)
Verifiable reward (math/code): ~$0.00
LM judge (Qwen3-8B, self-hosted): ~$0.003
LM judge (GPT-4o API): ~$0.08
At G=16, you call the judge 16x per prompt per step
In practice, most RLVR setups combine these: a format check (free) + a correctness verifier (cheap) or LLM judge (moderate). The eval cost is almost always dwarfed by rollout generation — except when code sandboxes have heavy test suites.
Multi-turn rollouts: how to account for eval cost
Single-turn tasks (math, competitive programming) have a clean cost model: rollout generation dominates, eval is a rounding error. But as tasks become multi-turn and environment-heavy, eval cost grows from negligible to dominant — and the simple tokens × $/token model breaks down.
The key variables that change:
Tokens per trajectory
A math rollout: ~512 tokens. A 20-turn SWE trajectory: 15K–50K tokens (tool outputs + reasoning). A thorough SWE-bench agent: up to 2M tokens. The rollout cost multiplier is 30–4,000x.
Sandbox time per trajectory
Each turn hits the environment. If a single sandbox eval takes 5s and you have 20 turns, that's ~100s of sandbox time per trajectory — and you're running G of them in parallel per prompt. The sandbox bill scales as turns × eval_time × G × batch_size × $/sandbox-second.
GPU sandboxes multiply the effect
When the reward needs a GPU (KernelBench, NanoGPT), you're paying GPU-hours for both the rollout model and the eval sandbox. 32 parallel A100 sandboxes at $2.80/hr = $89.60/hr just for eval. This can exceed the training GPU cost.
# Total eval cost per step
────────────────────────────────────────────────────────────
# Single-turn (math, G=16, batch=8)
128 rollouts × 1ms × $0/hr = $0.00
# Single-turn code (LeetCode, G=16, batch=8, Modal CPU)
128 rollouts × 10ms × $0.15/hr = $0.00005
# Multi-turn SWE (20 turns, G=4, batch=8, Modal CPU)
32 trajectories × 20 turns × 5s × $0.15/hr = $0.13
# CUDA kernels (4 turns, G=16, batch=8, Modal GPU)
128 trajectories × 4 turns × 3s × $2.80/hr = $1.19
# NanoGPT speedrun (1 eval, G=4, batch=4, 8xH100)
16 trajectories × 10min × 8 × $3.50/hr = $74.67
────────────────────────────────────────────────────────────
These are per-step costs — multiply by total training steps for the full bill
For the cost calculator, we model eval cost as a separate line item with three inputs: sandbox type (none / CPU / GPU), eval time per trajectory, and number of turns. For single-turn math and code tasks, the default is zero — you won't even see it. For multi-turn and GPU-sandbox tasks, it surfaces as a first-class cost component alongside rollout and training.