Training Quality
Numerical precision can silently corrupt RL gradients. Here's what goes wrong and how to prevent it.
The training-inference mismatch
When you call an inference API, you care about throughput and cost. Quantization (INT8, FP8, even FP4) is great — it makes tokens cheaper and faster. A few bits of precision loss doesn't matter when you're generating chat completions.
But in RL training, precision matters enormously. The policy update depends on comparing the current policy's log-probabilities against a reference policy's log-probabilities. If the rollout engine (inference) uses different precision than the training engine, those log-probs diverge — and the gradient signal becomes noisy or wrong.
This is the training-inference mismatch: the rollout phase runs inference (generating completions), but unlike a chat API, the numerical fidelity of that inference directly affects training stability.
Why the effect is so large: two compounding mechanisms
The mismatch is not just a small constant error — it grows as training progresses and as sequences get longer. Two mechanisms compound to cause catastrophic failures:
1. Error multiplies with sequence length
Log-probability of a sequence is the sum of per-token log-probs. Each token accumulates a small rounding error; over 8,192 tokens, those errors accumulate into a large systematic bias. The policy ratio used in the RL objective is an exponential of that sum — so multiplicative error compounds exponentially. "Beyond Precision" (2025) derives that gradient error scales as T² (quadratically with sequence length). QuRL (ICLR 2026) measures KL divergence growing 12× after 1,000 RL steps. At the extreme, the policy ratio can reach 10⁵ — causing gradient explosion if unclipped.
2. RL weight updates are tiny — quantization noise swamps the signal (precision mismatch)
In SFT, weight updates are on the order of 10⁻³ per step. In RL, they are 10⁻⁷ to 10⁻⁶ — two to three orders of magnitude smaller. INT8 quantization error is |θ|/2⁸, which is on the order of 10⁻³ for typical weight magnitudes. The quantization noise is literally 1,000× larger than the training signal. The model appears frozen — the quantized copy never moves in weight space despite the optimizer running. This is why INT8 rollout that works fine for SFT causes complete collapse in RL.
3. Token clipping silences rare but critical reasoning steps (multi-step IS)
PPO and GRPO stabilize multi-step training by clipping the per-token importance ratio to [1−ε, 1+ε]. But when a token's ratio moves outside that range, it receives zero gradient — it is completely silenced. This creates a subtle failure mode for reasoning tasks: rare but important tokens like key intermediate steps that the model initially assigns low probability may never receive a gradient signal, because every sample puts their IS ratio outside the clip range. The model cannot learn to produce those reasoning steps more reliably. This is why GSPO and DAPO use sequence-level rather than token-level objectives — they avoid this "token dropping" problem entirely. (Source: RLHF Book Chapter 6, GSPO section; DAPO paper)
The empirical consequence:
| Setup | BF16 (baseline) | Naive INT8 rollout | With correction |
|---|---|---|---|
| PPO, 0.5B — GSM8K (easy, short) | 55.35% | 48.78% | 53.55% |
| GRPO, 1.5B — 5 reasoning tasks | 56.40% | 52.31% | 55.48% |
| DAPO, 7B — AIME 2024 (hard, long chains) | 31.67% | 0.001% | 31.25% |
Source: QuRL (ICLR 2026). Corrections use Adaptive Clipping Range + Update-Aware Quantization.
The AIME result is the key data point: a 7B model trained with naive INT8 rollout essentially never learns on hard tasks. It's not a 5% penalty — it's a complete training collapse. The effect is length-dependent: Jet-RL (2025) finds BF16-train + FP8-rollout is tolerable at 4K tokens, meaningfully degrades at 8K, and diverges at 16K tokens. Exactly the regime where RLVR on complex tasks lives.
Temporal staleness vs numerical mismatch — two separate problems
PipelineRL's in-flight weight updates solve a different problem: temporal staleness, where rollout data was generated by a policy that is N optimizer steps behind the current training policy. This is orthogonal to numerical mismatch. A system can eliminate temporal staleness entirely (via in-flight updates) and still suffer catastrophic numerical mismatch if rollout runs in FP8 and training computes log-probs in BF16. Both effects independently corrupt the RL gradient — and they can compound.
BF16 vs FP16: a few bits change everything
A recent paper "Defeating the Training-Inference Mismatch via FP16" (Qi et al.) identifies the root cause: BF16's large rounding errors break consistency between training and inference policies.
| Format | Exponent bits | Mantissa bits | Decimal precision | Implication |
|---|---|---|---|---|
| BF16 | 8 | 7 | ~2.4 digits | Wide range, low precision. Log-prob rounding errors accumulate. |
| FP16 | 5 | 10 | ~3.3 digits | 3 extra mantissa bits → 8x less rounding error on log-probs. |
| FP32 | 8 | 23 | ~7.2 digits | Gold standard but 2x memory. Used for loss accumulation. |
The paper's finding: switching from BF16 to FP16 yields more stable optimization, faster convergence, and stronger performance across diverse tasks, algorithms, and frameworks — with only a few lines of code change. No architecture or algorithm modifications needed.
Why this matters for cost:
If BF16 rollouts cause unstable training, you waste steps (and tokens, and money) on noisy gradients. FP16 rollouts converge faster → fewer steps → lower cost. The precision "upgrade" is actually free (same memory footprint) and saves money through faster convergence.
The other quality axis: off-policyness
Precision mismatch isn't the only thing that corrupts gradients. Async RL introduces temporal staleness — rollouts generated by policy πold (k optimizer steps ago) training policy πcurrent. The rule of thumb: k ≤ 8 is generally safe with importance sampling correction; beyond that, variance explodes. These are independent axes — you can have perfectly fresh weights (k = 0) and still suffer precision mismatch if rollout runs in FP8 and training recomputes in BF16.
For the full treatment of staleness, the async spectrum (sync → tail batching → PipelineRL → replay), and how agentic tasks make it worse, see Efficiency Techniques: Staleness.
How frameworks handle it
The precision mismatch is especially dangerous when training and inference run in separate processes or engines. Each RL framework handles this differently:
- SLiME (Megatron + SGLang) — tight integration keeps training and inference on the same precision path. Powers GLM-5/4.7.
- veRL (FSDP/Megatron + vLLM/SGLang) — 3D-HybridEngine avoids the train→infer weight transfer that can introduce rounding.
- OpenRLHF (Ray + DeepSpeed + vLLM) — separate processes for training and inference. Must ensure matching precision config across Ray workers.
- Unsloth — custom Triton kernels with "0% loss in accuracy" claim. Single-process design avoids the multi-engine mismatch entirely.
- Tinker — managed platform handles precision internally. You don't configure BF16/FP16 — Tinker owns the full train+inference stack.
If you're stitching together TRL + vLLM yourself, the BF16/FP16 mismatch is exactly the kind of subtle bug you'll hit. The fix from Qi et al. is simple (switch to FP16), but you have to know to apply it — and across both your training and inference configs.
For a full comparison of framework abstraction levels, see The Abstraction.
Beyond precision: quantization for rollout
The flip side: aggressive quantization (FP8, FP4) during rollout can work if handled correctly. The QeRL paper shows that quantization noise during rollout generation actually aids exploration — acting as a form of regularization. The key distinction:
Split-precision mismatch (bad)
Training in BF16, rollout in FP8/INT8 — the two engines compute different log-probs for the same token sequence. Quantization noise dwarfs the RL gradient signal. Catastrophic on long sequences.
Unified quantization (can be good)
Both training and rollout use FP8 (Jet-RL approach) — zero precision gap. Or: rollout in FP4 to generate tokens faster, but log-probs recomputed at full precision during the training step. Quantization only affects the sampling, not the gradient calculation.
This is why Tinker's B200 FP4 rollout optimization from the 100x stack works: the quantized model generates completions faster, but the policy gradient is still computed at full precision during the training phase.