Efficiency Techniques
Two axes of improvement: use fewer samples, and make each sample cheaper.
RLVR cost = (tokens per step) × (steps to converge) × (cost per token). Efficiency techniques attack different terms in this product. We organize them into two categories:
Data Efficiency
Reduce the number of samples or steps needed to reach the same quality. Fewer rollouts per step, reusing old data, better learning signals.
Compute Efficiency
Reduce the cost per sample — faster rollout generation, overlapping phases, better hardware utilization. The innovation stack below.
These are multiplicative. A 4x data-efficiency gain (fewer steps) combined with a 25x compute-efficiency gain (faster per-step) yields 100x total cost reduction.
Data Efficiency
Techniques that reduce how many rollout tokens you need to reach your target quality. These save money regardless of your hardware or systems stack.
FilteringStop wasting tokens on noise
30-50% token savings
In standard GRPO, many rollout batches produce zero or near-zero gradient signal — either all G rollouts succeed (nothing to learn) or all fail (no positive examples). Filtering these out before training saves tokens without hurting convergence.
Zero-variance filtering: Skip prompts where all G rollouts get the same reward. With no variance across completions, the advantage is zero for every sample — pure noise in the gradient. ScaleRL uses this by default.
Adaptive difficulty filtering: Drop prompts with ≥90% pass rate (already mastered) and ≤5% pass rate (too hard to learn from). Focus training tokens on the edge of competence where the model actually improves. DAPO and Dr. GRPO both implement variants of this.
Together, these can reduce effective training tokens by 30-50% with no quality loss — you're just throwing away the tokens that weren't teaching the model anything.
Baseline & Variance ReductionBetter baselines, lower gradient variance
~1.5x convergence
GRPO's key insight: eliminate the critic network entirely by using the group mean reward as a baseline. This is far more compute-efficient than PPO (no value network to train, no extra forward passes) and simpler to implement. But the group-mean baseline has higher variance than a well-trained critic — some gradient steps correct for estimation noise rather than improving the policy. These refinements reduce that variance while keeping GRPO's critic-free simplicity.
RLOO (REINFORCE Leave-One-Out) — for each rollout, the baseline is the mean reward of all other rollouts in the group. This per-sample baseline has lower variance than the group mean, since it excludes the sample being evaluated. Converges faster with the same group size.
Dr. GRPO — removes the length normalization bias in GRPO that inadvertently rewards shorter responses. By fixing the advantage estimate, each gradient step moves in a more accurate direction. Also incorporates adaptive difficulty filtering.
DAPO — dynamic sampling: overproduces rollouts, filters to keep only informative ones, and uses clip-higher (asymmetric clipping) to prevent entropy collapse. ~1.5x faster convergence than GRPO.
GRPO
1x
group mean baseline
RLOO / Dr. GRPO
~1.5x
lower-variance baseline
DAPO
~1.5x
+ adaptive filtering
Alternative ObjectivesBeyond REINFORCE
stable async training
Standard REINFORCE/GRPO breaks down when rollouts are stale (off-policy) or when the policy collapses to a narrow mode ("entropy collapse"). These alternative objectives are designed to remain stable under off-policy data — critical for async pipelines where generation and training overlap — and to avoid the coverage collapse that kills output diversity on hard prompts.
CISPO — Clipped IS Policy Optimization
Clips the IS ratio but applies it as a coefficient (with stop-gradient) rather than clipping the objective directly like PPO. All tokens contribute gradients (no suppression in long sequences), giving ~2x faster convergence than GRPO with a higher performance ceiling. See Convergence & Scale RL for the interactive comparison.
DRO — Direct Reward Optimization (Kimi K1.5 / K2)
Replaces the clipped IS ratio with a squared KL penalty: L = log p(y)·A - ½β(log p/q)². The quadratic term provides smooth regularization without the cliff edges of PPO clipping. Naturally stable for off-policy data — used in Kimi K2's large-scale agentic RL training where async pipelines make staleness unavoidable. Available as a built-in loss in Tinker (loss_fn="dro").
OAPL — Optimal Advantage Policy Learning (Databricks KARL)
Regression-based objective: min (β log π/π_ref - advantage)². No importance weighting, no clipping. The key property: stable at 400+ gradient steps of policy lag, enabling deeply async training pipelines that would destabilize GRPO or PPO.
MaxRL — Maximum Likelihood RL
Maximizes the likelihood of producing at least one correct answer per prompt, rather than expected reward. This weights hard prompts more instead of marginalizing them, avoiding the coverage collapse where the model forgets how to solve rare problem types.
The common thread: these objectives stay stable when rollouts are stale and preserve output diversity under long training. This is what enables the async pipelines discussed below — you can't pipeline aggressively with vanilla GRPO. Tinker supports importance_sampling, ppo, cispo, and dro as built-in loss functions.
ArchitectureMoE base models: 1/6 the RL compute of dense
6x cheaper
ScaleRL found that an MoE model (17B×16 experts) reaches a higher ceiling than a dense 8B model while using only 1/6 of the RL compute. The base model carries more pre-trained capacity into RL, so each step extracts more signal — starting with a stronger base model can matter more than any algorithmic trick.
Compute Efficiency
Techniques that reduce the wall-clock and cost per step. RLVR spends ~85% of time on decode (autoregressive rollout generation), ~10% on training, ~5% on prefill. Decode is memory-bandwidth bound, so most gains come from reducing decode cost — but once decode is fast, training becomes the bottleneck (Amdahl's law).
Staleness: the async quality tradeoff
Async RL and Pipeline RL speed things up by overlapping generation and training — but that means rollouts are generated by an older policy than the one being trained. How stale is too stale?
| Task type | Rollout time | Optimizer steps during wait | Effective staleness |
|---|---|---|---|
| Math (512 tok) | ~1s | 0 | k = 0 |
| Code (single-turn, 24K tok) | ~45s | 1-2 | k = 1-2 |
| Agentic SWE (25 turns, sandbox) | ~5 min | 5-15 | k = 5-15 |
| Long agentic (50+ turns, complex env) | ~20 min | 20-60 | k = 20-60 |
For math, staleness is a non-issue. For agentic tasks with 5-minute rollouts, the policy moves 5-15 optimizer steps while a single rollout generates — and the slowest 20% of rollouts take 25-32x longer than the median, making their staleness far worse than the average. This is why staleness management is essential for scaling RL beyond math benchmarks.
Staleness: the core tradeoff
Staleness is the gap between the policy that generated a rollout and the policy that learns from it. In synchronous RL, staleness is zero: you generate rollouts, update the weights, then generate new rollouts with the updated weights. The data always matches the current policy.
What happens in one sync RL step:
1. Generate rollouts with policy πv=5
2. Score rollouts (reward model or verifier)
3. Compute policy gradient using πv=5 log-probs
4. Update weights → πv=6
5. Wait for step 1 to finish with new weights before training again
The wait in step 5 is where GPUs sit idle. For math tasks (512-token completions), this wait is seconds. For agentic tasks (25+ turns, 5 minutes per rollout), the wait is the entire rollout duration — GPUs are rented but doing nothing while sandboxes execute.
The temptation is obvious: while batch N trains, start generating batch N+1 with the current weights. By the time batch N+1 finishes, the weights have moved — the rollouts are now stale. The question is: how stale is too stale?
Measuring staleness: version lag k
If a rollout was generated by policy version v but trained on by version v + k, its staleness is k. The importance sampling ratio r = πv+k(a|s) / πv(a|s) corrects for this — but as k grows, these ratios become large and high-variance, eventually destabilizing training. The Effective Sample Size (ESS) = 1/E[r²] measures how much useful signal remains: ESS=1.0 is fresh, ESS<0.3 is mostly noise.
Off-policyness: how far can you go?
Async RL introduces temporal staleness. The rollout was generated by policy πold (k optimizer steps ago), but the training step updates πcurrent. How large can k get before quality degrades?
How off-policyness is measured
The standard tool is the policy ratio r = π(a|s) / πold(a|s), and its aggregate: Effective Sample Size (ESS) = 1/E[r²], ranging from 0 (fully off-policy) to 1 (perfectly on-policy). PPO clips r to [1−ε, 1+ε] (typically ε = 0.1–0.2) to limit how stale the data can be. The equivalent measure in async RL is the lag k: how many optimizer steps behind is the rollout policy.
The “k ≤ 8” rule of thumb
Empirically, lag k ≤ 8 optimizer steps is generally safe with standard importance sampling corrections. Beyond k ≈ 8–10, the IS correction weights become large and noisy — variance explodes and gradient quality degrades. VCPO (2025) specifically identifies that high-lag async training (k = 10–12) causes variance blowup and proposes ESS-guided step-size scaling to stabilize it — achieving 2.5x speedup on AIME-2025 without the instability.
| Approach | Typical lag k | ESS | Quality verdict |
|---|---|---|---|
| Synchronous PPO/GRPO (G=1) | 0 | 1.0 | Baseline — perfectly on-policy |
| Synchronous GRPO (G=16) | 0–1 | ~0.9 | Standard — within-batch slight staleness |
| PipelineRL (in-flight weights) | <1 (sub-step) | ~0.75 | No observed degradation |
| Async RL (AReaL, Laminar) — moderate | 2–8 | 0.5–0.8 | Safe with IS correction |
| High-lag async (k = 10–12) | 10–12 | <0.4 | Variance blowup without VCPO-style correction |
For how replay buffers improve data efficiency by reusing old rollouts, see the data efficiency section above. Note: temporal lag and numerical precision mismatch are independent axes — see Training Quality for the precision side.
The solution spectrum
There is no single best approach — each trades off hardware efficiency (GPU idle time) against data quality (how on-policy your gradients are). The right choice depends on how long your rollouts take.
Key insight: PipelineRL achieves near-async throughput (6 steps) with near-synchronous data freshness. Instead of syncing weights between batches, it swaps them mid-sequence at token boundaries. Early tokens carry some lag, but the critical final reasoning tokens — where the reward signal comes from — are always generated under the latest policy.
Synchronous (k = 0) is the baseline: generate, train, generate, train. Zero staleness, but GPUs idle during every generation phase. Fine for math RL where rollouts take seconds. Wasteful for anything longer.
Tail batching stays synchronous but sorts prompts by expected length — short rollouts run together (fast rounds), long ones are batched separately (slow rounds). No staleness, no IS correction needed. RollPacker reports 2-2.5x over veRL with this alone.
Double buffering (k = 1): while training on batch N, start generating batch N+1. One step of staleness — within the PPO clip range, minimal quality impact. Used by veRL, SLiME, verifiers-rl.
Pipeline RL is the most impactful approach. Synchronous RL has a bubble problem: during rollout (~70-85% of wall-clock), training GPUs sit idle; during training, generation GPUs sit idle. And within rollout, the slowest sequence gates the entire batch. PipelineRL (ServiceNow) fixes this by maintaining a constant pool of active sequences per GPU — when any sequence finishes, it's immediately replaced. The key innovation is in-flight weight updates: the trainer pushes updated weights to generation workers mid-sequence without halting generation.
Key insight: PipelineRL achieves near-async throughput with near-synchronous data freshness. Instead of syncing weights between batches, it swaps them mid-sequence at token boundaries. Early tokens carry some lag, but the critical final reasoning tokens — where the reward signal comes from — are always generated under the latest policy. ESS stays at ~0.75, comparable to synchronous G=8.
On 128 H100s: ~2x wall-clock speedup to same final quality. VCPO (MIT Han Lab) extends this with ESS-guided learning rate scaling, pushing to 2.5x speedup on AIME-2025 (42h vs 105h synchronous). AReaL (Ant Group) takes a different approach — interruptible rollouts that discard stale KV caches when new weights arrive, achieving 2-2.3x at 512-GPU scale. ROLL Flash uses queue-based scheduling to dispatch completed responses immediately (2.2x, 3.4x generation-only).
Bounded async queues (k = 2-K) go further: multiple batches in flight with a version bound. Stale trajectories are either IS-corrected or simply aborted — RollArt aborts any trajectory more than 1 version behind and launches redundant rollouts to compensate. VCPO is stable to k = 128 with its variance-controlled correction.
Replay buffers are a research direction worth watching. The idea: instead of discarding rollouts after one gradient step, store them and replay alongside fresh data — providing contrast when all current rollouts get similar rewards. RePO and RLEP show promising results on math benchmarks, but no production RL system uses replay buffers yet. The interaction with importance sampling at scale, and whether the gains hold for noisier agentic tasks, remains unvalidated.
Practitioner note: one-off pipelining is often good enough
Apparently, simple double-buffering (k=1) or synchronous training is the pragmatic choice for most teams. Getting a good model is what matters most, and the additional complexity of deep async pipelines introduces failure modes that are hard to debug. The problems async brings (stale gradients, IS correction tuning, harder reproducibility) are often not worth the throughput gains — especially since performance also depends heavily on data quality and task design, not just pipeline efficiency.
RL training is fragile: collapse is often unrecoverable
Unlike pre-training where you can usually restart from a checkpoint, RL training is uniquely vulnerable to reward collapse — once the model begins showing signs of entropy collapse or reward hacking, it is almost impossible to recover the model. Rolling back to an earlier checkpoint rarely helps because the data distribution has shifted. This makes aggressive async pipelining especially risky: stale gradients can push the model into a collapse regime before you detect it.
For a comprehensive survey of how 16 open-source RL libraries handle async training, see "Keep the Tokens Flowing" (HuggingFace, 2026).
Hardware reliability & goodput
At cluster scale, hardware failures are not edge cases — they are the steady state. RL training is uniquely vulnerable because the generation-training loop means failures waste hours, not minutes.
Hardware failures at scale
Meta's Llama 3.1 training report documented 419 unexpected interruptions over 54 days on 16,384 H100s — roughly one failure every 3 hours. 58.7% were GPU-related (GPU faults, NVLink errors, HBM3 memory failures).
Meta's longer-term reliability study (11 months, 16K A100s, 150M+ GPU-hours) established a scaling law: MTTF is proportional to 1/Ngpus.
| Scale | MTTF | Source |
|---|---|---|
| 8 GPUs | 47.7 days | Meta RSC study |
| 1,024 GPUs | 7.9 hours | Meta RSC study |
| 3,000 GPUs | 33 hours | Nebius production cluster |
| 16,384 GPUs (Llama 3.1) | ~3 hours | Meta Llama 3.1 report |
| 100,000 GPUs | ~30 min | SemiAnalysis estimate |
Why RL training is uniquely vulnerable
Pre-training has a simple failure mode: the job crashes, you restart from the last checkpoint, and you lose a few minutes of work. RL training is worse because of the generation-training loop:
Generation is the long pole
RL spends 50-80% of wall-clock on rollout generation. A single batch of rollouts on a 32B model with 32K-token outputs can take hours. If a failure occurs during generation, the entire batch of in-flight rollouts is wasted — not just the few minutes since the last checkpoint, but potentially hours of generation work.
Sync RL amplifies the blast radius
In synchronous RL, all GPUs must complete their rollouts before training can proceed. A single node failure stalls the entire cluster. RobustRL reports >3,000 machine-induced interruptions per week at 100K GPU scale.
Async RL as natural resilience
Disaggregated architectures (separate generation and training pools) provide natural fault tolerance: if a generation node fails, the inference pool continues on remaining nodes while the failed node recovers. This is a key reason async RL achieves 25-68% faster end-to-end than synchronous — not just from overlap, but from resilience.
Useful training time vs. wasted time
A simple way to think about this: what fraction of your wall-clock time actually produces useful training steps, versus time lost to failure recovery, checkpointing, or idle waiting? This ratio determines your actual cost, not peak TFLOPS or raw MFU.
| Recovery strategy | Recovery time | Goodput | Wasted cost* |
|---|---|---|---|
| Naive (manual detect + replace) | ~2.5 hrs | ~89% | $5.6M |
| Automated (Llama 3 / MegaScale) | ~45 min | ~96% | $1.7M |
| Hot spares + fast checkpointing | ~minutes | >95% | $0.3M |
| torchforge + CoreWeave (measured) | — | 96% | — |
*Wasted cost modeled on a $54M, 50-day run on 4,000 H100s with 1 failure/day. Source: The Data Scientist: From FLOPs to Goodput
For most researchers self-hosting RL on 8-64 GPUs, hardware failures are infrequent enough to manage manually. But the cost of a single failure — losing hours of in-flight rollouts — is proportionally larger because your runs are shorter. A managed platform like Tinker absorbs this entirely: you pay per token, and tokens lost to infrastructure failures are Tinker's problem, not yours.