Empirical Validation
Grounding the cost model in real training runs.
The cost calculator models RLVR economics from first principles. But do the predictions hold? This page runs quick, cheap training jobs to ground-truth the cost model against actual Tinker bills.
These aren't serious training runs — GSM8K is a saturated benchmark and a 4B model is small. The point is to validate that our token accounting and cost formulas match reality before applying them to larger, more expensive experiments.
Quick Validation: Tinker Cost on GSM8K
GRPO with Qwen3-4B-Instruct on GSM8K (7.5K problems). One epoch, 59 steps, LoRA rank 32, 128 groups × 8 rollouts = 1,024 sequences/step. Total cost: $12.96.
Cost model vs actual bill
| Predicted | Actual (W&B) | ||||
|---|---|---|---|---|---|
| Billing meter | Rate | Tokens | Cost | Tokens | Cost |
| Sample (rollout gen) | $0.22/M | 34.0M | $7.49 | 24.7M | $5.44 |
| Train (full seq fwd+bwd) | $0.22/M | 34.0M | $7.49 | 30.0M | $6.60 |
| Prefill (no caching) | $0.07/M | 6.0M | $0.42 | 13.5M | $0.95 |
| Total | 74.1M | $15.39 | 68.3M | $12.99 | |
| Actual Tinker bill | $12.96 | ||||
Actual bill: $12.96. Predicted from token counts: $12.99. Error: 0.2%. (Cost model formula overestimates at $15.39 due to assumed 512 avg completion vs actual 320.)
How we got actual tokens: summed from W&B metrics logged by the Tinker cookbook every step. Sample = training rollout completions (19.3M from env/all/total_ac_tokens) + eval rollouts (5.4M from test/env/all/total_ac_tokens, 12 eval runs × 1,319 test problems). Train = prompt + completion per training datum (total_ob_tokens + total_ac_tokens = 10.7M + 19.3M). Prefill = all prompt tokens (training + eval). The Tinker console reports “billable tokens” = sample + train + prefill combined, which matches our W&B sum within 0.1% for both experiments.
How we got predicted tokens: the cost calculator formula: Sample/train = batch (128) × group_size (8) × avg_completion (512) × steps (59) × length_growth_avg ((1+1.2)/2). Prefill = steps × batch × group_size × avg_prompt (100) = 6.0M (no prefix caching — each rollout pays full prompt). The model doesn't account for eval rollouts, and overestimates completion length (512 vs actual 320).
Key insight: The cost model overestimates completion length (predicted 512, actual avg 320) but doesn't count eval tokens. These errors roughly cancel out, landing within 0.2% of the actual bill. A more accurate model would use actual completion length and explicitly include eval cost.
Confirmed: no prefix caching
Each rollout in a group of 8 shares the same prompt (fewshot prefix + problem). If Tinker cached the shared prefix across the group, we'd see far fewer prefill tokens. Instead, every rollout pays for the full prompt. Our cost formula correctly assumes no caching.
| With caching | No caching (predicted) | Actual (measured) | |
|---|---|---|---|
| Prefill tokens / step | ~40K | ~181K | ~181K |
| Total prefill (59 steps) | ~2.4M | ~10.7M | 10.7M |
| Prefill cost (@ $0.07/M) | $0.17 | $0.75 | $0.75 |
With caching, the 8 rollouts per group would share the prompt prefix (~128 unique prompts/step × 178 tok). Without caching, all 1,024 rollouts pay full prompt cost (1,024 × 178 tok). The actual prefill matches the no-caching prediction exactly.
Impact at scale — cache-aware billing is planned
For Qwen3-4B at $0.07/M prefill, the overcharge is only $0.58 per run — negligible. But the overcharge scales with G × model price. On a 32B model ($0.49/M prefill) with 25-turn agentic rollouts (swe_rl config), the same per-rollout billing produces $34.6K in prefill vs $4.3K with per-group caching — a $30K difference that dominates the total cost.
Tinker's sequence extension docs confirm this: “Sampling infrastructure doesn't yet adjust billing based on KV-cache hits, but this is planned for a future release.” When this ships, multi-turn agentic workloads on Tinker would become dramatically cheaper.
What we validated
- Cost prediction is accurate. $12.99 predicted from token counts vs $12.96 actual bill (0.2% error). The three billing meters (sample, train, prefill) account for every dollar.
- No prefix caching. Prefill tokens/step match the no-caching prediction exactly (~181K). Each of the 1,024 rollouts pays for the full prompt including the shared fewshot prefix. Our cost formula correctly assumes this.
- Train tokens = full sequence. Tinker bills train tokens on prompt + completion (30M), not just completions (19.3M). The cost model must account for this.
- Eval rollouts are billed. The 12 test evaluations (1,319 problems each) added 5.4M sample tokens to the bill. Running evals more frequently increases cost — worth considering for longer runs.
What would this cost self-hosted? (8x B200)
veRL exp1a ran the same GSM8K task on 8x B200 GPUs. Here's how the costs compare:
| Tinker | veRL (8x B200) | Cost model predicted | |
|---|---|---|---|
| Wall-clock | 33 min | 1h 56m | ~28 min (MFU=10%) |
| Training MFU | n/a | 1.6% | 10% (RL default) |
| Compute cost | $12.96 | ~$77 | ~$19 |
| Setup time | ~0 | ~1.5 hrs | not modeled |
| GSM8K accuracy | 91.3% | 93.1% | — |
The cost model now defaults to MFU=10% (RL with proper offloading), predicting ~$19. Actual with our broken offloading (MFU=1.6%): ~$77 — a 4x gap from the bug alone. With proper CPU offloading, self-hosted cost would be ~$19-25, closer to the model's prediction.MFU remains the most important parameter for self-hosted cost.
Tinker vs Self-Hosted veRL on Hendrycks MATH
The cost calculator tells you what training should cost. But what does it actually feel like to run the same job on a managed API vs your own GPU cluster? We ran GRPO on Hendrycks MATH with Qwen3-4B-Instruct-2507 on both Tinker and an 8x B200 cluster to compare: setup friction, training quality, step time, systems insights, and final cost.
The veRL self-hosted runs were conducted on FluidStack B200 GPUs at the SemiAnalysis x FluidStack hackathon. Thanks to FluidStack for providing Blackwell compute access. The veRL run crashed at step 35 of 94 when the hackathon cluster shut down, but we have enough data to compare.
Results
| Tinker (94 steps) | veRL (35/94, crashed) | |
|---|---|---|
| MATH-500 accuracy | 78.9% | 79.2% (at step 35) |
| Avg step time | ~81s | ~119s (growing to 237s) |
| Avg response length | 714 tok (stable) | 684 tok avg (549 → 1376) |
| Response clip ratio | n/a | 44% at crash |
| Total wall-clock | ~127 min | ~70 min (35 steps) → ~190 min projected |
| MFU | n/a (managed) | 1.85% |
| Total ac tokens | 68.5M | ~24M (partial) |
veRL run crashed at step 35 of 94 — the hackathon cluster was shut down. Both platforms used Qwen/Qwen3-4B-Instruct-2507 (same model, fair comparison).
Tinker cost: predicted vs actual
| Predicted | Actual (W&B) | ||||
|---|---|---|---|---|---|
| Billing meter | Rate | Tokens | Cost | Tokens | Cost |
| Sample (rollout gen) | $0.22/M | 92.4M | $20.33 | 75.1M | $16.53 |
| Train (full seq fwd+bwd) | $0.22/M | 92.4M | $20.33 | 86.9M | $19.12 |
| Prefill (prompt tokens) | $0.07/M | 1.8M | $0.13 | 20.2M | $1.42 |
| Total | 186.6M | $40.78 | 182.3M | $37.06 | |
| Actual Tinker bill (console diff) | $37.01 | ||||
W&B token accounting: $37.06. Actual bill: $37.01. Error: 0.1%.
Actual cost and tokens from the Tinker console (“billable tokens” = sample + train + prefill). W&B token sums match the console within 0.1%, confirming all three billing meters are accounted for. Cost model overestimates by 10% ($40.78 vs $37.01) — same pattern as exp1a: overestimates completion length (assumed 800, actual 714) but doesn't count eval tokens (19 eval runs added 6.7M sample + 1.8M prefill). Errors partially cancel. Effective rate: $0.203/M tokens (vs $0.190/M in exp1a).
No prefix caching (confirmed again)
| With caching | No caching | Actual | |
|---|---|---|---|
| Prefill / step | ~42K | ~196K | ~196K |
| Total prefill (94 steps) | 4.0M | 18.4M | 18.4M |
| Cost savings if cached | $1.01 (not applied) | ||
MATH prompts avg 191 tok (vs 178 on GSM8K). Confirmed per-rollout billing — same result in exp1a.
How we verified this
Step 1: W&B token counts. The tinker-cookbook logs two metrics per step: env_all_total_ac_tokens (action/completion tokens) and env_all_total_ob_tokens (observation/prompt tokens). Summing across 94 steps: 68.5M ac + 18.4M ob.
Step 2: Compute predicted bill. Using Tinker's listed rates ($0.22/M sample, $0.22/M train, $0.07/M prefill): 75.1M sample × $0.22 + 86.9M train × $0.22 + 20.2M prefill × $0.07 = $37.06.
Step 3: Compare to Tinker console. The Tinker console (the source of truth for billing) reports 182.0M billable tokens and $37.01. Our prediction: 182.3M tokens, $37.06. Match within 0.1%.
Step 4: Per-rollout vs per-group. ob_tokens/step = ~196K. Per-rollout prediction (B×G×prompt = 128×8×191) = 196K ✓. Per-group prediction (B×prompt = 128×191) = 24K ✗. If Tinker deduplicated within groups, billable tokens would be ~16M lower and the bill ~$1 less. They're not — the console total matches the per-rollout model exactly.
Tinker docs context. Tinker's sequence extension docs note: “Sampling infrastructure doesn't yet adjust billing based on KV-cache hits, but this is planned for a future release.” This is about multi-turn KV reuse, not explicitly about within-group prefix sharing — but it's consistent with our finding that billing doesn't reflect caching.
With prefix caching pricing: prefill would drop from 18.4M to 2.3M tokens (8x reduction = G), saving $1.01 on this run. Small here ($1.01 of $6.96), but the overcharge is proportional to G × prompt_length × steps × $/M — on larger models with longer multi-turn prompts, this becomes the dominant cost. See the takeaway for the swe_rl projection ($34.6K → $4.3K prefill with caching).
Exp 1b vs Exp 1a
| Exp 1a (GSM8K) | Exp 1b (MATH) | Ratio | |
|---|---|---|---|
| Total cost | $12.96 | $37.01 | 2.9x |
| Total tokens | 68.1M | 182.0M | 2.7x |
| Avg completion | 320 tok | 714 tok | 2.2x |
| Steps | 59 | 94 | 1.6x |
| Effective $/M | $0.190 | $0.203 | 1.07x |
Cost scales linearly with token volume. The 2.9x increase comes from 1.6x more steps × ~1.7x more tokens/step (longer completions + longer prompts). The slightly higher $/M is because MATH has proportionally more train tokens (longer prompts increase full-sequence train cost relative to sample cost).
Why does veRL training on this cluster have low MFU?
This is not normal for RL training, but it is expected for our specific setup. The low MFU is caused by colocated architecture + a CPU offload bug workaround:
- Colocated = memory contention. veRL runs the actor model and vLLM rollout engine on the same 8 GPUs. They timeshare GPU memory — when vLLM generates rollouts, the actor model sits idle (but still occupies VRAM). This forces smaller training micro-batches and wastes GPU cycles.
- CPU offload bug. veRL normally offloads the actor to CPU during rollout to free GPU memory for vLLM. A bug in the restore path (
param_offload=Falseworkaround) means the actor stays on GPU at all times — the worst case for memory contention. - RL pipeline has 5 serial phases. Even without memory issues, the gen → logprob → ref → train → weight-sync pipeline has idle gaps between phases. SFT does one forward-backward; RL does ~5 passes per step.
Expected MFU by setup:
Is this normal? What the literature says
Yes — low MFU is well-documented for colocated RL training. Multiple recent papers confirm that 80-90% of RL training time is spent on rollout generation (memory-bandwidth bound, low arithmetic intensity), leaving the compute-bound training phase as a small fraction of wall-clock.
- RollMux (Dec 2025): Phase-level multiplexing for disaggregated RL. Tested on 328 H20 + 328 H800 GPUs. 1.84x cost efficiency over standard disaggregation by multiplexing rollout and training across jobs.
- RollArt (Dec 2025): Disaggregated agentic RL infrastructure. 2.05x speedup over colocated baselines on 3,000+ GPUs.
- StreamRL (2025): Disaggregates training and generation, running them asynchronously. Rollout benefits from high memory-bandwidth inference GPUs.
Can multi-tenancy help?
Yes — managed platforms like Tinker can multiplex multiple LoRA jobs on shared base models, keeping GPUs busy while individual jobs do rollout. This is fundamentally why per-token pricing can be cheaper than renting GPUs: you don't pay for idle time between phases. See Tinker Multiplexing for how this works.
How is MFU measured?
veRL reports perf/mfu/actor = estimated FLOPs / (peak BF16 TFLOPS × wall-clock) / num_GPUs, measured only during the update_actor phase (not the full RL step). So the 1.85% means even the training phase itself has poor GPU utilization. LoRA makes this worse: the backward pass flows gradients through the full 4B model but only updates ~0.5% of parameters (the LoRA adapters). Combined with small micro-batches forced by memory contention, the GPUs are mostly moving data, not computing.
Training curves
Tinker (orange, 94 steps complete) vs veRL (blue, 35 steps before crash).
MATH-500 test accuracy. Both converge to ~79% by step 35. Tinker continues to 78.9% at step 93 (accuracy plateaus — harder problems don't yield to more training).
Key findings
- Quality parity again. Both platforms reach ~79% MATH-500 accuracy by step 35. The platform doesn't determine quality.
- Response length explosion on veRL. veRL responses grew 2.5x (549 → 1376 tok) with 44% hitting the 2048 cap. Tinker stayed stable at ~714 tok. Same model, same task, and neither platform applies a length penalty (reward is purely correctness + format). The difference is likely the 49x LR gap: Tinker uses 4.9e-4 (from their LoRA-specific LR recommendation — 10x base LR for LoRA), while veRL uses a generic 1e-5 not tuned for LoRA. The higher LR converges faster to concise correct answers. With no length penalty in the reward (just correctness + format), GRPO has no signal to shorten responses — so the slow veRL LR lets the model drift verbose before converging.
- Length growth is the cost driver on hard tasks. On GSM8K (exp1a), response length was stable. On MATH, it can double. The cost model's
lengthGrowthFactoris validated as critical for harder tasks. - Step time scales with response length. veRL step time tripled (76s → 237s) as responses grew. Tinker step time was stable because its responses didn't grow.
- MATH accuracy plateaus. Tinker accuracy peaked around step 30 (~86%) then fluctuated down to 78.9% by step 93. More training doesn't help — the 4B model has a ceiling on competition math.
Experiment setup details
Configuration
Both platforms use Qwen/Qwen3-4B-Instruct-2507 with LoRA rank 32. Max tokens bumped to 2,048 (vs 1,024 in exp1a) for longer MATH reasoning chains. The cookbook combines both Hendrycks “train” and “test” splits for training (~12.5K total, minus 500 MATH-500 = ~12K). This is standard practice.
Tinker (managed)
| Model | Qwen/Qwen3-4B-Instruct-2507 | Same as exp1a |
| Learning rate | 4.9e-4 | get_lr() recommended for LoRA |
| KL coef | 0.0 | Cookbook default |
| Loss function | importance_sampling | |
| Batch config | 128 groups x 8 rollouts | |
| Infrastructure | Tinker API (managed) | |
| Steps/epoch | ~94 | ceil(12000/128) |
veRL (self-hosted)
| Model | Qwen/Qwen3-4B-Instruct-2507 | Same model as Tinker |
| Learning rate | 1e-5 | veRL default for GRPO |
| KL coef | 0.001 | low_var_kl |
| Loss function | GRPO (adv_estimator=grpo) | |
| Batch config | batch_size=128, n=8 | |
| Infrastructure | 8x B200 (FluidStack), FSDP2 + vLLM | Crashed at step 35 (cluster shutdown) |
| Steps/epoch | ~94 | ceil(12000/128) |
veRL setup (8x B200, FluidStack)
Total setup time: ~1.5 hours. Three bugs fixed in local veRL checkout:
setnot subscriptable in FSDP2 wrap policy — changed toNone not in- flash-attn fallback — can't compile on CUDA 13.1 + torch cu128. Added try/except to use
transformersfallback - CPU offload restore bug —
param_offload=Falseworkaround. Actor stays on GPU during vLLM rollout, causing the low MFU
Stack: veRL 0.8.0.dev0, torch 2.10.0+cu128, vLLM 0.17.1, 8x NVIDIA B200 192GB. See experiments/exp1a/NOTES.md for full bug details and patches.
Note on veRL stability
veRL has been experiencing stability issues in recent versions, making it difficult to use as a reliable self-hosted baseline for benchmarking. For future experiments, we are considering using SLiME as the self-hosted comparison framework instead, given its production-proven stability at scale (powers GLM-5/4.7). This would also let us test Megatron-based training as the alternative to Tinker's managed approach.
Planned experiments
- Exp 1b: Complete — Hendrycks MATH. See above. Key finding: response length growth is the cost driver on hard tasks.
- Exp 2: Scale to Qwen3-8B and Qwen3-30B-A3B — tests the cost model's MoE pricing assumptions.
- Exp 3: Async RL on Tinker vs sync baseline (SLiME or veRL) — tests whether multi-tenancy and async scheduling change the convergence curve.
- Exp 4: Colocated vs disaggregated — same model, same data, but separate rollout GPUs + training GPUs. Isolates the MFU penalty from memory contention.