FLOP Accounting
The naive calculation, and why it's misleading.
Assumptions on this page
- Full fine-tune (6N FLOPs/token for training) — LoRA savings discussed separately below
- Colocated — rollout and training share the same GPUs (not disaggregated)
- Fully on-policy — fresh rollouts every step, no replay buffer or off-policy reuse
The formulas
For one RLVR step with B prompts, G rollouts per prompt, L average completion tokens, K gradient epochs, and Pactive active parameters:
| Component | Formula | Character |
|---|---|---|
| Rollout | B x G x L x 2 x Pactive | Memory-bandwidth-bound |
| Ref model logprobs | B x G x L x 2 x Pactive | Compute-bound (parallel prefill) |
| Training (backward) | K x B x G x L x 6 x Pactive | Compute-bound |
| Optimizer step | O(Ptrainable) | Memory-bandwidth-bound (not compute) |
The 2N vs 6N paradox
Training costs 3x more FLOPs per token than rollout (6N vs 2N). A naive analysis predicts training dominates cost. But FLOPs ≠ time, because of the utilization gap:
Rollout (decode)
Bandwidth-bound. Reads all weights from HBM per token, one matrix-vector multiply. MBU is decent but MFU is terrible — tensor cores sit idle waiting for data.
Training (backward)
Compute-bound. Large matrix-matrix multiplies across full batches. Tensor cores stay busy — MFU is the right metric here.
Note that the optimizer step itself is also memory-bandwidth-bound, not compute-bound — it reads and writes optimizer states (Adam moments) but does minimal arithmetic. This further reduces the effective MFU of the training phase, since the optimizer update adds wall-clock time without adding useful FLOPs.
The key insight: rollout and training are bottlenecked by different resources, so comparing them via FLOPs alone is misleading. Rollout speed is set by memory bandwidth (MBU ~60%), not compute. Training speed is set by tensor core throughput (MFU ~35%). Converting to wall-clock: a rollout token at ~3% MFU (bandwidth-bound decode) costs 2N ÷ 0.03 ≈ 67N FLOPs worth of GPU-time. A training token at ~35% MFU (compute-bound) costs 6N ÷ 0.35 ≈ 17N FLOPs of GPU-time. The "cheaper" operation is actually ~4x more expensive per token in wall-clock.
Interactive: throughput by model and hardware
Decode throughput (bandwidth-bound)
209 tok/s /GPU @ MBU=100%
838 tok/s effective (8 GPU, ~70% MBU)
Train throughput (compute-bound)
86,625 tok/s (8 GPU, 35% MFU)
Hardware reality check (roofline model)
Can the hardware config above actually support 1,024 concurrent rollouts? The roofline model answers whether decoding is compute-bound or memory-bandwidth-bound, and whether the KV cache fits in VRAM.
Decode regime
?Above B_crit → tensor cores are the bottleneck.compute-bound
B_crit (crossover point)
?B_crit = GPU_TFLOPS / GPU_bandwidth = 990 / 3.35 ≈ 296. Below this batch size, decode is bandwidth-bound. Above it, compute-bound. Your config: 1024 seq/GPU.296 seq/GPU
KV cache (rollout)
?KV cache for all 1024 concurrent sequences. Model weights: ~38 GB. Total VRAM: 640 GB. Fits comfortably (max ~7343 seqs).83.9 GB
Arithmetic intensity
?FLOPs per byte of memory accessed during decode. At batch=1024, intensity ≈ 1024 FLOP/byte. The crossover (B_crit) is 296 FLOP/byte. You're above it → compute-bound.~1024 FLOP/byte
VRAM utilization
?38 GB model weights + 83.9 GB KV cache = 122 GB used out of 640 GB total VRAM.~19%
LoRA FLOPs savings
?LoRA training: ~4N FLOPs/token (forward + activation gradients only). Full fine-tune: 6N FLOPs/token. The 33% savings comes from skipping weight gradients on frozen base layers — but this shifts the cost balance even further toward rollout dominance.33% fewer train FLOPs
Length growth cost
?As training progresses, the model produces longer reasoning chains. Completions grow from ~512 to ~768 tokens (1.5× growth). Budget for this when estimating total rollout cost.+20% more tokens
Per-step breakdown by sequence length
B=64, G=16, Qwen3-8B (8B active) on 8x NVIDIA H100 80GB SXM. Each row is one RL step — rollout FLOPs use the 2N rule, train FLOPs use 6N, but wall-clock time depends on utilization.
| Seq len | Tokens/step | Rollout FLOPs (2N per token) | Train FLOPs (6N per token) | Rollout time (low MFU) | Train time (high MFU) | Rollout % (of wall-clock) |
|---|---|---|---|---|---|---|
| 512 | 0.5M | 8.4 PFLOP | 25.2 PFLOP | 5.3s | 6.1s | 47% |
| 1,024 | 1.0M | 16.8 PFLOP | 50.3 PFLOP | 14.3s | 12.1s | 54% |
| 2,048 | 2.1M | 33.6 PFLOP | 100.7 PFLOP | 43.3s | 24.2s | 64% |
| 4,096typical | 4.2M | 67.1 PFLOP | 201.3 PFLOP | 2.4m | 48.4s | 75% |
| 8,192 | 8.4M | 134.2 PFLOP | 402.7 PFLOP | 8.7m | 1.6m | 84% |
| 16,384 | 16.8M | 268.4 PFLOP | 805.3 PFLOP | 33.1m | 3.2m | 91% |
Key takeaway
At a typical 4096-token completion: rollout has 3x fewer FLOPs than training (67.1 PFLOP vs 201.3 PFLOP), yet takes 3.0x longer in wall-clock (2.4m vs 48.4s). Rollout is 75% of step time because autoregressive decoding runs at ~3% GPU utilization vs ~35% for training.
What this means for pricing
On Tinker, rollout vs training cost is transparent: you pay separately for sample (rollout) and train tokens. On self-hosted GPU-hours, the rollout inefficiency is hidden inside a single hourly bill where you can't tell what you're paying for.
As sequence length increases, the rollout fraction climbs — from 47% at 512 tokens to 91% at 16,384 tokens. The longer the reasoning chains, the more RLVR looks like pure inference.
Model architecture choices that improve rollout throughput
KV cache size is one of the main constraints on rollout throughput, because it limits how many sequences you can decode concurrently. Architectural innovations that compress KV cache have an outsized impact on the phase that dominates cost.
DeepSeek V3 uses Multi-head Latent Attention (MLA), which compresses KV cache by 93.3%. This enables ~6x more concurrent sequences during rollout — a massive throughput multiplier for the phase that dominates cost. Combined with MoE's active param efficiency (37B active out of 671B total), it makes V3 surprisingly cost-effective for RLVR despite its 671B total size.
In practice, most modern models already use MLA or GQA — these are the baseline, not the exception. Standard multi-head attention is largely a thing of the past for models targeting efficient inference. The relevant comparison is between GQA and MLA (both efficient) as the floor, with linear attention as a potentially cheaper alternative. Linear attention variants (e.g. RWKV, Mamba-style state-space models) replace the quadratic attention computation entirely, offering O(1) per-token KV “cache” and potentially even higher rollout throughput — though at a quality tradeoff that's still being characterized for RL.