Takeaway
What makes managed per-token training possible, and when it makes sense.
1. LoRA + MoE make per-token pricing possible
LoRA's many-to-one base-to-adapter ratio means hundreds of users share one copy of the base weights. MoE gives more model capacity per FLOP. Together: shared weights, tiny per-user state, inference on active parameters only — the prerequisites for multi-tenant GPU sharing.
2. The advantage is multiplexing, not FLOP savings
LoRA saves some FLOPs — but the real value is multiplexing: different users in different RL phases share the same GPU pool. Tinker fills the idle gaps between phases with other users' work. Co-serving training and inference on the same silicon pushes utilization from ~30% to ~70%.
3. Multiplexing creates hard RL-specific problems
On-policy RL needs bitwise-reproducible logprobs, but multi-tenant batches change size every cycle. The shared base model ops (matmul, RMSNorm, attention) produce different results at different batch sizes. Batch-invariant kernels fix this at a performance cost — the per-user SGMV adapter path is less affected since each user's segment is already isolated. The training quality page covers the broader precision and staleness challenges.
4. The abstraction separates data efficiency from compute efficiency
RLVR efficiency breaks into two halves: data efficiency (how much reward signal you extract per sample) and compute efficiency (how fast the hardware processes those samples). Tinker's four primitives (sample, forward_backward, optim_step, save_state) expose the data-efficiency knobs you actually want to experiment with: loss function (PPO, CISPO, DRO, or fully custom via forward_backward_custom), advantage estimation, KL penalty shaping, and filtering logic. Meanwhile Tinker absorbs the compute-efficiency side: off-policiness numerics (importance ratios, batch-invariant kernels), async pipelining, kernel fusion, memory management, and multi-tenant scheduling. You focus on the algorithm; Tinker handles the rest.
5. The cost model grounds all of this in real numbers
We built a FLOP accounting framework, a cost calculator comparing per-token vs GPU-hour pricing, and empirical validation against real Tinker and veRL runs. Convergence and eval infrastructure complete the picture.
Where Tinker wins — and where it doesn't
Workloads inspired by tinker-cookbook recipes and published RL training setups. Self-hosted assumes a well-optimized setup: SGLang (70% MBU), 1-step async (k=1), 8x H100 SXM at $2.69/hr. Tinker doesn't win on short-horizon math RL — a well-run veRL setup is cheaper there. But RL runs are inherently asynchronous: rollout, reward, and training happen in phases with idle gaps. The longer the horizon, the more those gaps compound — and that's where Tinker's multiplexing pays off.
| Workload | Model | Tinker | Self-hosted | Winner |
|---|---|---|---|---|
| math_rl180 steps, 512 tok | Qwen3-8B | $99 | $28 | Self-hosted (3.5x) |
| code_rl100 steps, 24K tok, sandbox | Qwen3-4B | $1,665 | $2,110 | Tinker (1.3x) |
| harbor_rl100 steps, 20K tok, agentic | Kimi-K2 (1T MoE) | $617 | $997 | Tinker (1.6x) |
| swe_rl200 steps, 25-turn agentic | Qwen3-32B | $38,733 ($6,885 w/ prefix caching) | $7,903 | Self-hosted (4.9x) |
| osworld_rl180 steps, 30-turn GUI VMs | Qwen3-8B (VLM proxy) | $4,035 ($635 w/ prefix caching) | $2,609 | Self-hosted (1.5x) |
math_rl, code_rl, harbor_rl are inspired by tinker-cookbook recipes — same task structure but adapted models (Qwen3 instead of Llama) and realistic completion lengths (cookbook defaults are placeholders). harbor_rl uses the cookbook's exact model (Kimi-K2) and group_size (4). swe_rl from DeepSWE/SkyRL, osworld_rl from ComputerRL (arXiv 2508.14040). Self-hosted: 8x H100 SXM @ $2.69/hr, SGLang 70% MBU, async k=1. All numbers from the cost calculator — try different configs yourself.
The pattern: Tinker wins on workloads with sandbox idle time and large models. Harbor RL on Kimi-K2 (1T MoE) is 1.6x cheaper on Tinker because self-hosted needs many GPUs just to load the model, and those GPUs sit idle during sandbox execution. Code RL is similar — Tinker's per-token pricing avoids paying for idle phases.
Self-hosted wins on math RL (short rollouts, high GPU utilization, small model) and on SWE RL with heavy multi-turn prefill. We confirmed empirically (exp1a & exp1b) that Tinker bills prefill per rollout, not per group — G=8 rollouts sharing the same prompt pay the prompt 8 times. In a 25-turn trajectory, the growing conversation history is re-billed at every turn. Most of the $38.7K SWE RL Tinker cost is prefill. Self-hosted with KV caching (92% hit rate) doesn't pay for this. Tinker's docs confirm cache-aware billing is planned but not yet live.