Convergence & Scale RL
How many steps do you actually need, and how do you know when to stop?
RLVR cost breaks into four mostly-independent pieces: inference (rollout token generation), training (forward-backward on those tokens), convergence (how many steps until performance saturates), and environment (sandbox cost for reward verification). The first two are well-understood — as an API consumer paying per token, the question is just “how many tokens?” The fourth is environment-dependent. But convergence — when to stop training — is where the real uncertainty lives.
RL saturates — the sigmoid scaling law
Pre-training loss follows power laws (unbounded, always improving with more compute). RL accuracy doesn't — it's bounded between 0 and 1, and it saturates. ScaleRL (Meta, 400K+ GB200-hours of experiments) showed RL post-training follows a sigmoidal curve. Values below are converted to H100-equivalent GPU-hours (×2.8 factor — GB200 has 2.4× the bandwidth and 4.5× the compute of H100):
A asymptote
Performance ceiling. Set by loss function and numerical precision — not by how long you train.
B exponent
Compute efficiency. Higher = faster ascent to ceiling. Set by infrastructure and algorithmic details.
Cmid midpoint
GPU-hours at 50% of total gain. The halfway point of your training run.
R0 baseline
Starting performance. Set by the base model before any RL.
Compare two algorithms
Different clipping strategies shift different parts of the sigmoid. Some raise the ceiling. Some speed up convergence. The best recipes do both. See algorithm reference for memory profiles, FLOP formulas, and implementation details.
Symmetric clipping (eps=0.2), sample-level loss. Baseline recipe from DeepSeek-R1-Zero.
Truncated IS with stop-gradient — all tokens contribute gradients (no suppression in long sequences). ~2x faster convergence than DAPO.
Tinker estimate
$27.8K
Self-hosted estimate
$80.7K
ScaleRL config assumptions:
Tinker = sample + train tokens (Llama-3.1-8B pricing, no sandbox). Self-hosted = GPU rental only (8× H100 @ $2.69/hr, no engineering overhead). See full calculator for custom configs.
GRPO (DeepSeek recipe)
CISPO (MiniMax-M1)
What you can control
These map to two kinds of efficiency gains — see Efficiency Techniques for the full breakdown.
Raises the ceiling (A) — data efficiency
Higher ceiling = more performance per token of RL. These let you reach a better final result with the same compute budget.
- •Loss function: CISPO → A=0.61 vs GRPO → A=0.50 (+22%). How you clip importance weights determines what the model can learn.
- •FP32 at logits: +0.09 absolute in ScaleRL — essentially free. Fixes inference-training precision mismatch.
- •Longer generation: 32K → higher A than 14K. More thinking tokens = higher ceiling.
- •Larger batch: Small batches look better early but plateau lower on downstream evals.
- •Zero-variance filtering: Skip prompts where all G rollouts get the same reward — no gradient signal, just wasted tokens.
- •Adaptive filtering: Drop prompts with ≥90% pass rate — already solved, wasting tokens on easy problems.
Speeds up convergence (B) — compute efficiency
Same asymptote, faster wall-clock arrival. These don't change what you learn — just how quickly you get there.
- •PipelineRL: 4x throughput via async — same asymptote, faster wall-clock arrival. The DualPipe of post-training.
- •Async RL: Separate GPU pools for generation and training — eliminates idle time between phases.
- •Prompt-level loss: Average per prompt, not per sample — stabilizes gradients, fewer wasted steps.
- •Hardware: B200 (2.4x bandwidth) and Vera Rubin (6.6x) directly speed up the decode bottleneck.
- •Sequence packing: Eliminates padding waste in training — 3-10x more useful tokens per batch.
Your base model matters most
The base model determines R0 (where you start) and strongly influences how fast you converge. Three findings stand out:
MoE models need ~1/6 the RL compute
ScaleRL's 17Bx16 MoE (Llama-4 Scout) reached the 8B dense model's ceiling using only 1/6 of the RL compute — and then kept improving past it. More total parameters means more latent capability for RL to surface. For Tinker cost: an MoE model may converge in fewer steps, saving tokens even though per-token cost is similar.
RLVR surfaces existing capability, not new reasoning
“Limit of RLVR” (NeurIPS 2025) showed that RLVR improves sampling efficiency — getting the right answer in fewer attempts — but doesn't create reasoning capability that isn't already latent in the base model. A math-specialized base will converge faster because the capability is already there; RL just amplifies it.
SFT cold-start trades speed for cost
A short SFT phase before RL (DeepSeek-R1 style) accelerates initial convergence by giving the model the right output format. But SFT+RL requires nearly 2x the total training time of RL-from-scratch because SFT induces longer responses. Open-Reasoner-Zero achieved comparable results to R1-Zero with 1/10 the steps using vanilla PPO on Qwen2.5-32B base — suggesting a strong base model can skip cold-start entirely.
Framework choice: same steps, different wall-clock
Frameworks (veRL, OpenRLHF, TRL, torchforge) implement the same algorithms, so identical hyperparameters should yield identical convergence trajectories in steps. The difference is wall-clock time: a framework with 2x throughput halves the cost to reach the same performance, but doesn't change how many steps you need.
The exception: numerical precision differences between frameworks can silently change the asymptote. ScaleRL's FP32-at-logits fix (+0.09 absolute) is a precision detail that varies by framework. If your framework recomputes log-probs in a different precision than rollout generation, the importance sampling ratio corrupts — and your ceiling drops without any obvious signal in the training curve.
Async RL (PipelineRL, VCPO) is particularly interesting: it changes B (convergence speed) without changing A (ceiling). ScaleRL found k=8 off-policy steps to be optimal — VCPO achieved 2.5x faster wall-clock on AIME-2025 (42h vs 105h) with the same final accuracy. For Tinker users, async scheduling is handled internally — you get the throughput benefit without managing it.
Task difficulty
Convergence speed depends heavily on where the task sits relative to the model's current ability. RLVR only works at the “edge of competence” — problems the model can solve with non-random accuracy but hasn't mastered.
| Factor | Faster convergence | Slower convergence |
|---|---|---|
| Task type | Math (dense reward, exact-match) | Agentic/SWE (sparse reward, multi-turn) |
| Reward density | Binary correct/incorrect (clear signal) | Sparse pass/fail over long trajectories |
| Difficulty curve | Smooth spectrum (relay effect) | Discontinuous (grokking-like plateaus) |
| Group size (G) | G ≥ 8 (stable advantages) | G = 2-4 (high variance, but cheaper per step) |
| KL penalty | Low or zero (more exploration) | High (>0.04 degrades quality by over-constraining) |
Reference convergence data
One practical approach to estimating convergence: pick an adjacent task where data exists.
| Task | Model | Steps | Compute | Source |
|---|---|---|---|---|
| MATH / GSM8K | 8B | ~180 | ~3,500 GPU-hrs | tinker-cookbook, SimpleRL-Zoo |
| MATH (ScaleRL) | 8B | ~7,400 | 100K GB200-hrs (~280K H100-eq) | ScaleRL |
| MATH (ScaleRL) | 17Bx16 MoE | ~7,100 | 50K GB200-hrs (~140K H100-eq) | 1/6 compute of 8B for higher perf |
| AIME-24 | 7B | ~1,500 | ~27h × 8 A100 | Dr. GRPO (43.3%) |
| Reasoning (R1) | 671B MoE | ~8,000 | ~$1M | Epoch AI |
| SWE-Bench | 32B | ~100-200 | — | DeepSWE / SkyRL-Agent (24.4% → 39.4%) |
Projecting where you'll end up: the 25% rule
The most economically useful finding from ScaleRL: fit the sigmoid on the first 25% of training and predict where the run ends up. This turns convergence estimation from guesswork into calibration:
- Run a short pilot (25% of your expected compute budget)
- Fit the sigmoid to your reward curve — estimate A, B, Cmid
- Decide: is the predicted ceiling worth the remaining 75%?
- If not, adjust the recipe (switch loss function, fix precision, change generation length) and re-pilot
Cmid is the key number here. It's the GPU-hours where you've captured 50% of the total gain (visible as the dot on each curve in the chart above). If your pilot reaches Cmid, you have high confidence in the sigmoid fit — you can see both the slope and the inflection, which pins down A and B. The earlier you can estimate Cmid, the earlier you can decide whether to commit or pivot.
This is the RL analogue of Chinchilla-style compute-optimal planning. The sigmoid makes it possible to be rational about RL investment.
What this means for scaling up RL
Whether you're on Tinker or self-hosted, convergence determines how many steps you actually run — and that drives your total cost. The practical levers:
- Pick the right base model. An MoE model may converge in 1/6 the steps of a dense model. A math-specialized base converges faster on math. RL surfaces existing capability — it doesn't create it.
- Match task difficulty to model capability. RLVR only works at the edge of competence. Too easy (all rollouts succeed) or too hard (none succeed) = wasted tokens. Filter accordingly.
- Pilot first. Run 25% of your compute budget, fit the sigmoid, and decide if the projected ceiling justifies the remaining 75%.
- Algorithm matters. CISPO can reach GRPO's ceiling in half the tokens. Switching loss functions is free.
- Generation length trades tokens for ceiling. 32K context raises A but each step costs more — find the sweet spot for your task.
- LoRA doesn't change the curve. Thinking Machines showed LoRA fully matches full fine-tuning in policy gradient RL, even at rank 1. RL learns ~1 bit per episode — far less capacity than LoRA provides.
For efficiency techniques that reduce cost per step, and the cost calculator to estimate total tokens — multiply steps (this page) by tokens per step (calculator) for your expected spend.
Note: The sigmoid parameters in the chart above are illustrative, based on relative performance ratios from ScaleRL. Absolute numbers depend on model, dataset, and hardware — the relative ordering and shape is what matters. Important caveats on ScaleRL: (1) Their results are validated primarily on math and coding benchmarks — the sigmoid scaling law may not generalize to other domains. (2) Their runs only go up to ~2K steps; the fitted formula has not been validated beyond that range, so extrapolating to 8K+ steps should be treated with caution. (3) Most RL algorithms tend to reach similar asymptotic performance given the same number of training items (unique problems seen) — the differences in the chart above are largely about wall-clock efficiency, not fundamental capability ceilings. See also Rishabh Agarwal's Scaling RL guest lecture for a deeper dive on this topic. See also RL for LLMs (Aweers, 2026) for a comprehensive walkthrough of GRPO, PPO, and the algorithms compared above.