Every RLVR training iteration decomposes into three phases with starkly different compute profiles. Understanding this breakdown is the foundation of any cost model.

The inversion

A useful rule of thumb: at production scale, you typically need ~6 sampler workers to saturate 1 trainer — the generation bottleneck is that severe.

This is the core insight: the phase with fewer FLOPs per token dominates the cost. Rollout generation costs 2N FLOPs/token versus 6N for training — yet it consumes 80-90% of wall-clock time. The reason is utilization: autoregressive decoding reads all model weights for each token but only performs a single matrix-vector multiply, achieving single-digit percent of peak TFLOPS. Training parallelizes across tokens and achieves 30-50% of peak.

The Laminar framework paper confirms this empirically: up to 83.1% of total execution time is spent on generation for reasoning tasks.

The reward signal spectrum

The cost breakdown above assumes Phase 2 is cheap. That's true for textbook RLVR on math — checkers are essentially free. But even “verifiable” rewards can be expensive. Code generation and SWE tasks require spinning up Docker containers, running test suites, and waiting for execution — far more costly than an LLM-as-judge rubric call. A SWE-bench evaluation averages ~3.5 minutes per instance. The term “verifiable” says nothing about cost.

More fundamentally, the reward function is part of an environment integration — like an OpenAI Gym environment. The right abstraction isn't “reward function as a component of the RL framework” — it's the environment that encapsulates task setup, execution, observation, and reward. Frameworks that treat reward functions as a first-class internal concern are conflating levels of abstraction. In practice, the reward comes from one of three places:

The trend is clear: as you move from verifiable to learned rewards, Phase 2 goes from negligible to a dominant cost center. And there's a hidden cost that none of these categories capture: data and environment creation. Building the rubrics, curating preference pairs, setting up sandboxed execution environments, writing unit tests — this is engineering and domain-expert time that doesn't show up in GPU-hours but can easily dominate the total project cost for non-math/code tasks.

How frameworks handle reward model deployment

In practice, most RLVR tasks use verifiable rewards (math checkers, unit tests, format validators) — so the reward model placement question rarely dominates system design. When you do need a learned RM or LLM-as-judge, the question becomes: where does it live?

At scale, the dominant approach is disaggregated deployment — separate GPU pools for generation, training, and reward scoring. Colocated setups that time-multiplex GPU memory (only one of {generation, RM, training} occupies VRAM at a time) are mainly used for small-scale runs where provisioning separate pools isn't worth the overhead. Note that colocated time-multiplexing is inherently synchronous — you can't overlap phases on the same GPUs.

At large scale, disaggregated deployment (separate GPU pools per component) is the default — it's simpler to scale and enables fully async pipelines where generation, scoring, and training overlap. OpenRLHF exemplifies this. Colocated approaches like veRL's sleep/wake, SLiME's offload, and NeMo's CPU swap share a common idea — only one phase occupies GPU memory at a time — but these are primarily useful for small-scale runs (≤8B models on a few GPUs) where the overhead of managing separate pools isn't justified.

For LLM-as-judge (when it's needed — most RLVR tasks don't require it), the dominant pattern is an external inference endpoint. veRL's GenRM and OpenRLHF's --remote_rm_url both decouple the grader from training GPUs entirely — you call an inference API behind an HTTP endpoint, and the training loop sends async requests. This is architecturally identical to how tinker-cookbook's rubric recipe works: the grader LLM runs on Tinker's inference service, not on the training GPUs.

This matters for cost modeling. LLM-as-judge is an external inference cost — the grader model is hosted elsewhere (an API, a separate inference cluster), so it doesn't appear in your training GPU budget at all. You pay per-token to the inference provider, and that cost scales with group size and rubric count, but it's a line item on a different bill. A learned reward model is a training infrastructure cost — it sits in your VRAM (colocated) or on GPUs you provision (disaggregated). Either way, it's GPU-hours you're paying for directly and must account for in capacity planning.

There's also a latency consideration that affects throughput. With verifiable rewards, scoring takes milliseconds — it never bottlenecks the pipeline. With a colocated RM, the forward pass adds wall-clock time to each step but runs at GPU speed. But with LLM-as-judge, the grading calls can take 2–30 seconds per rubric item, and even with async parallelism, this can become the pacing bottleneck for rollout evaluation. If your GRPO group size is 16 and you have 3 rubric items, you're waiting on 48 inference calls to complete before the policy update can start. At scale, the reward evaluation latency — not the reward compute cost — becomes the binding constraint on iteration speed.

The long-tail problem

During rollout, all GPUs generate completions in parallel. But completion lengths follow a long-tail distribution — the 99th percentile can exceed the median by an order of magnitude. Every GPU must wait for the longest completion to finish before the batch can proceed. This synchronization penalty is invisible in FLOP accounting but very real in wall-clock time.

Worse, response length grows during training — typically 1.5-3x over the course of a run as the model learns to produce longer reasoning chains. This makes rollout cost the single most volatile variable in the entire cost model.

Algorithm memory comparison

The choice of RL algorithm has a first-order effect on VRAM. The table below shows which model copies each algorithm requires in memory simultaneously. Source: RLHF Book (Lambert, 2026), Chapter 6.

The “reward model” column is where the reward signal spectrum matters most. With verifiable rewards (math, code), the RM is a deterministic verifier — GRPO needs only 2 model copies (policy + reference). With rubric-based grading, the grader LLM runs separately (via inference API, not in training VRAM) — still 2 copies for GRPO, but inference cost for the grader. With a learned reward model, you're back to 3+ copies. With LoRA, adapters add only ~100–500 MB per model on top of frozen weights, plus Adam optimizer states (~2× adapter size in FP32). This is why GRPO + LoRA + verifiable rewards fits on far fewer GPUs than PPO + full fine-tune + learned RM.

Multi-turn: the prefill multiplier

For single-turn tasks (math, code generation), the token accounting is straightforward: total inference tokens ≈ total training tokens. You generate B × G completions per step, each averaging L tokens, and then train on those same tokens. The prefill-decode ratio is fixed by your prompt length.

Multi-turn changes this significantly. Each turn requires a new prefill that includes the full conversation history — all prior turns' prompts and responses become the next turn's prompt. For a 10-turn trajectory:

The prefill tokens grow quadratically with turns — you're re-encoding the entire history at each step. For the harbor_rl workload (10 turns, ~20K trajectory), prefill accounts for ~60% of total inference tokens, compared to <5% in single-turn math. This is why multi-turn agentic tasks are disproportionately expensive: the model is “yappy” — generating many short responses that each require re-reading a growing context.

For cost estimation, the multiplier is roughly: total_inference_tokens ≈ decode_tokens × (1 + turns/2) for a uniform-length trajectory. As an API consumer paying per-token, this is the key number — you don't care about MFU or packing, you care about how many tokens flow through the system.

How much compute do I actually need?

Here are the three tinker-cookbook recipes sized across H100 and B200. These are the minimum GPU counts to fit the model (policy + reference + overhead) and the estimated wall-clock for a full run. Numbers from the cost model — plug in your own parameters in the calculator.

Min GPUs = minimum to hold policy + reference in VRAM (GRPO + LoRA, BF16, 1.4× overhead). Wall-clock uses 8 GPUs for math_rl/code_rl, min required for harbor_rl (Kimi-K2 at 1T params). B200 gains come from 2.4× higher bandwidth (faster rollout) and 4.5× higher TFLOPs (faster training).

A reference data point

Epoch AI estimates DeepSeek R1's RL phase at ~6.1 x 10²³ FLOPs — roughly 20% of V3's pretraining cost, or approximately $1M on H800 hardware. This gives a sense of RLVR costs at frontier scale: significant, but an order of magnitude cheaper than pretraining.