tinker-nomics

Cost Model

How the calculator estimates Tinker and self-hosted costs — assumptions, formulas, and known gaps.

The token breakdown

Every RLVR step generates three distinct streams of tokens, each billed (or computed) separately:

Rollout (sample)

B × G × L × S

64 × 16 × 512 × 180 = 94M tok

Completion tokens from autoregressive decode. Dominates cost at ~70–80% of total.

Training (forward-backward)

rollout × K

94M × 1 = 94M tok

Policy update tokens. K = gradient epochs (num_substeps). Default K=1 means same count as rollout.

Prefill

B × G × P × S

64 × 16 × 200 × 180 = 36.9M tok

Input tokens processed during sampling. Billed per rollout — each of the B×G rollouts pays for the full prompt. No prefix caching discount.

Where B = prompts per step, G = rollouts per prompt (group size), L = avg completion length, S = total steps, P = avg prompt length.

Length growth: completions lengthen as the model learns to reason. We model this with a growth factor (default 1.5×), averaging early and late lengths over the run: avg_multiplier = (1 + factor) / 2 = 1.25×. This tends to be the most volatile variable in the cost model — easy to underestimate in practice.

These formulas have four scaling dimensions. Training data size is usually fixed by your task. The levers you control — and the ones that matter for cost — are:

Samples per data point (G) — harder tasks need more rollouts to find reward signal
Rollout length (L) — longer sequences cost more per sample and reduce concurrency (each sequence consumes more KV cache, so fewer fit in memory, so GPU utilization drops)
Base model size — larger priors scale every other dimension proportionally

Task difficulty compounds G and L together. Latency sensitivity pushes the other way — if the trained model needs fast responses, you pick a smaller model or add length penalties to the reward.

Tinker pricing

Tinker charges three per-token rates for each model: prefill, sample (rollout), and train. The total API cost is:

cost = (rollout_tok × price_sample

+ train_tok × price_train

+ prefill_tok × price_prefill) / 1_000_000

Sandbox cost (Modal or SandboxFusion for code/agent tasks) is billed separately and added on top. Use SandboxFusion for zero-cost local execution.

No prefix caching discount: Tinker bills prefill per rollout (B × G × P per step), not per prompt. Even though Tinker may use prefix caching internally, users pay for all input tokens. This makes prefill cost scale with group size — at G=16 with long prompts, prefill can become a significant cost component. Confirmed empirically in Experiment 1a.

Self-hosted: rollout throughput

Autoregressive decode is memory-bandwidth bound at realistic batch sizes. Each generated token requires reading all model weights once from HBM (High Bandwidth Memory). The roofline model gives throughput as:

single_gpu_tps = bandwidth_bytes_per_s / model_size_bytes

// H100 + Qwen3-8B BF16: 3.35e12 / 16e9 = 209 tok/s per GPU

At large batch sizes, KV cache reads add overhead. The full batched formula:

bytes_per_step = model_bytes + kv_bytes_per_token × seq_len × batch

effective_tps = (bandwidth / bytes_per_step) × batch × efficiency

The rollout efficiency factor (0–1) captures variable-length padding waste, KV-cache overhead, and scheduling latency. Typical values: 0.30–0.50 for synchronous GRPO; 0.50–0.70 with async RL or a dedicated rollout pool.

There is a crossover batch size where decode flips from bandwidth-bound to compute-bound:B_crit = peak_FLOPs / bandwidth ≈ 295 for H100. For GRPO with B=64, G=16 → 1024 concurrent sequences, well above B_crit — so the batch is compute-bound if the inference engine parallelizes them all together (vLLM/SGLang do).

Self-hosted: training throughput

Training is compute-bound. The throughput formula:

train_tps = (num_gpus × peak_tflops × MFU) / flops_per_token

// LoRA: flops_per_token = 4N (not 6N for full fine-tune)

// MFU default: 0.10 (RL+LoRA with proper offloading; SFT = 0.35)

The 4N FLOPs/token for LoRA (vs 6N for full fine-tune) comes from skipping weight gradients for frozen layers — only activation gradients propagate backward through the frozen backbone. LoRA adapter gradients (~0.5% of params) are negligible.

FORWARD

All layers, frozen + LoRA

BACKWARD (activations)

Must flow through frozen layers to reach earlier LoRA adapters

BACKWARD (weights)

≈0

Only LoRA params (~0.5%) need weight gradients

Agentic multi-turn mode

For single-turn tasks (math, basic code), a flat avgCompletionLength works. But multi-turn agentic tasks (SWE-bench-style) have a fundamentally different cost structure: each turn adds tokens to a growing conversation context, creating quadratic prefill growth.

// Per-rollout: N turns, O output/turn, Obs observation/turn

output_per_rollout = N × O

raw_prefill = N × P + (O + Obs) × N × (N-1) / 2

// Quadratic in N — this is why agentic RL is expensive

The model accounts for failure overhead (failed rollouts run 18-82% more turns) and length growth over training. Key data sources:

PER-TURN TOKENS

~350 output, ~750 observation

mini-SWE-agent (arxiv 2511.02230): 13.7K assistant + 30.4K tool msgs over ~40 steps

AVG TURNS

18 → 25 over training

SkyRL (arxiv 2511.16108). RollArt reports 30-50 range for SWE-bench.

Tinker vs self-hosted prefill: Tinker bills per raw input token — you pay the full quadratic sum regardless of internal caching. Self-hosted can use prefix caching (SGLang/vLLM) to skip reprocessing unchanged context between turns. The cacheHitRate slider only affects self-hosted compute time, not Tinker cost.

Async off-policyness (k)

Synchronous RL wastes GPU time waiting for rollouts. The longer the rollouts, the worse the waste — agentic tasks with 5-minute rollouts can leave GPUs idle for 50-60% of wall-clock time. See Training Quality for the full staleness-efficiency tradeoff.

The cost model uses a single parameter k (off-policy staleness bound) to control how much generation and training overlap. This maps directly to Tinker's AsyncConfig(max_steps_off_policy=k).

k = 0 — Synchronous (on-policy)

1x baseline

Generate → train → generate → train. Zero staleness but GPUs idle during generation. Total wall-clock = rollout hours + training hours (sequential).

Tinker: Default (no AsyncConfig). All cookbook recipes use this by default.

k = 1 — Double buffer (1-step off-policy)

~1.7x

Generate batch N+1 while training on batch N. Eliminates tail waste and sandbox idle. One step of staleness — within PPO clip range. No abort overhead. Total wall-clock = max(rollout, training) instead of the sum.

Tinker: StreamMinibatchConfig(groups_per_batch=B, num_minibatches=N). Also: veRL, SLiME, verifiers-rl.

k ≥ 2 — Async queue (k-step off-policy)

~2-2.7x

Multiple batches in flight with a staleness bound. Rollouts older than k training steps are discarded and their prompts requeued with fresh weights. Higher k = more tolerant of staleness = fewer aborts. Requires IS-corrected losses (PPO, CISPO, DRO).

Tinker: AsyncConfig(max_steps_off_policy=k, groups_per_batch=B). Also: PipelineRL (in-flight weight updates, ~2×), RollArt (per-trajectory abort), VCPO (2.5× wall-clock at k=2, stable to k=128 with ESS-guided LR).

How we model it

The key insight: when k > 0, generation and training overlap. Wall-clock time becomes the bottleneck (slower phase), not the sum:

// Sync (k = 0): sequential

total_hrs = rollout_hrs + train_hrs

// Async (k > 0): overlapped

total_hrs = max(rollout_hrs, train_hrs)

The stale abort rate models the cost of discarded rollouts. Higher k means a more generous staleness bound, so fewer rollouts are aborted:

abort_rate = 0.20 / k // k ≥ 2; 0 for k < 2

abort_mult = 1 / (1 − abort_rate) // extra rollouts to compensate

sample_tok = B × G × abort_mult × steps × L

train_tok = B × G × steps × L × K // training uses full batch, no abort

Tinker cost is unaffected by k. Tinker handles async scheduling internally — its per-token pricing already absorbs pipelining efficiency. The abort rate and overlap savings only affect the self-hosted estimate.

When does higher k help most?

The overlap benefit depends on how balanced your workload is. For math_rl (0.5h rollout vs 1.3h training), k=1 already captures the full overlap benefit — training is the bottleneck regardless. For workloads where rollout ≈ training time, higher k gives the system more scheduling flexibility. For rollout-dominated workloads (agentic, harbor_rl with 90h rollout vs 5h training), the overlap benefit is small and k=1 is the sweet spot — the main win is eliminating sandbox idle time.

Replay buffers ( RePO, RLEP) can further improve convergence speed but are not yet modeled in the calculator — no widely-adopted framework has standardized replay for RLVR yet. See /quality for details.

VRAM and minimum GPUs

We estimate the minimum GPUs needed to hold the model in memory:

model_bytes = total_params × 2 bytes // BF16

total_bytes = model_bytes × 2 × 1.8 // 2 copies + 80% overhead

min_gpus = max(2, ceil(total_bytes / gpu_vram))

Why 2 copies, not 3? In colocated GRPO with LoRA, the actor and reference model share the same base weights — the reference is just the frozen base, and the actor applies LoRA adapters on top. So actor + reference = 1 copy of base weights. The vLLM rollout engine needs its own copy (merged base + LoRA) for inference. Total: 2 model copies.

For full fine-tuning (no LoRA), the reference would be a separate frozen copy, requiring 3 model copies + full optimizer states. The calculator assumes LoRA.

The 1.8× overhead covers: vLLM KV cache for rollout (~20-40 GB at batch=128×8), training activations (~10-20 GB with gradient checkpointing), LoRA adapter + optimizer states (~0.5 GB at rank 32), FSDP communication buffers, and memory fragmentation. Minimum 2 GPUs enforced for RL (vLLM needs at least TP=2).

These are theoretical minimums. In practice, veRL and OpenRLHF both recommend 8 GPUs for 8B colocated GRPO. Smaller GPU counts force tiny micro-batches (→ low MFU) and limit vLLM's KV cache (→ slow rollout). Our experiments used 8x B200 for a 4B model.

We also track the KV cache for the full concurrent rollout batch (B × G sequences at avg completion length) and flag when it exceeds available VRAM. The KV cache formula is:

kv_bytes_per_token ≈ active_params_B × 0.016 KB

kv_cache_GB = kv_bytes_per_token × batch × seq_len / 1e9

LoRA checkpoint size

We estimate checkpoint size from actual layer shapes (when architecture data is available), mirroring the logic in tinker-cookbook/tinker_cookbook/hyperparam_utils.py get_lora_param_count().

LoRA targets all 2D weight matrices except: MoE routing gates, embedding tables, and DeepSeek MLA's already-compressed q_b_proj /kv_b_proj. This includes both attention projections (q/k/v/o) and FFN layers (gate/up/down). The FFN contribution is ~2–3× larger than attention alone — which is why a naive "0.5% of params" heuristic (calibrated to attention-only) was off by 2× for most models.

Per-layer formula for dense GQA models:

dim_sum_per_layer =

2×d // q_proj [d, d]

+ (kv_dim + d) × 2 // k_proj, v_proj

+ 2×d // o_proj [d, d]

+ (ffn + d) × 3 // gate, up, down projections

lora_params = rank × num_layers × dim_sum_per_layer

checkpoint_GB = lora_params × 2 / 1e9 // BF16

For MoE models, we use attention + shared expert only (conservative lower bound). Routed expert FFN LoRA size is uncertain — depends on whether weights are stored as 2D or 3D tensors in safetensors.

Model	Checkpoint (rank=32)	Storage/month
Qwen3-8B	186 MB	$0.019/mo
Qwen3-32B	512 MB	$0.051/mo
Llama-3.1-8B	168 MB	$0.017/mo
Llama-3.1-70B	828 MB	$0.083/mo
Qwen3-235B-A22B	447 MB	$0.045/mo
Qwen3-30B-A3B	116 MB	$0.012/mo
DeepSeek-V3.1	~1.1 GB	~$0.11/mo
Kimi-K2-Thinking	~960 MB	~$0.096/mo

What the model does not cover

Activation memory during training

Backprop stores intermediate activations for all layers in the batch. At long sequence lengths this can rival model weight memory. E.g. Qwen3-32B with B=64, L=8192: ~220 GB of activations before gradient checkpointing. We don't track this — use gradient checkpointing for long-sequence training.

Long-tail rollout padding waste

Our rolloutEfficiency slider captures this as a scalar, but the actual penalty depends on the completion length distribution. A p99 / p50 ratio of 10× causes ~50% GPU idle time during the tail. The slider default (0.50) is a rough estimate. Hard reasoning tasks can require 10K–100K+ token completions (RLHF Book, Chapter 6), making this the dominant throughput bottleneck at scale.

Sequence packing

A key mitigation for long-tail waste not modeled here: sequence packing stacks multiple short completions into a single batch slot using attention masking. This fills the idle compute that would otherwise wait for long-tail sequences. Frameworks like vLLM and SGLang implement this; it effectively increases the rolloutEfficiency for mixed-length batches. We don't model it — treat rolloutEfficiency as a rough upper bound on what packing can achieve.

Communication overhead (multi-GPU)

FSDP/DeepSpeed all-reduce and all-gather add latency proportional to model size / network bandwidth. Not modeled; absorbed into the MFU assumption (0.10 for RL+LoRA vs 0.35 for SFT).

Sandbox execution latency

Code/agent tasks require running the generated code and waiting for results before the next step. This can add 1–10s per rollout. We model the dollar cost (Modal pricing) but not the wall-clock latency impact on throughput.

Reward model cost (RLHF)

This model targets RLVR (verifiable rewards — math, code). RLHF with a neural reward model adds a full forward pass per rollout, typically 10–30% extra cost. Not modeled.

Real-world reference: OLMo 3 32B

The RLHF Book Appendix C documents the AllenAI OLMo 3 32B training run — one of the few public post-training cost breakdowns from a serious open lab:

SFT SWEEP

~36 hrs × 256 GPUs

per learning rate; 4 LRs tested

DPO SWEEP

~18 hrs × 64 GPUs

per LR; repeated over multiple days

INITIAL RL RUN

~5 days

"at least a day lost to instability"

CONTINUED RL (OLMo 3.1)

21 days × 224 GPUs

≈ 113,000 GPU-hours

Key observation: recipe development (finding the right hyperparameters, data mix, etc.) costs 10–100× the final training run. The calculator estimates a single run — multiply accordingly for the full R&D budget.

Source

The model is implemented in two places that are kept in sync:

Python: tinker_nomics/cost_model.py, tinker_nomics/hardware.py
TypeScript: web/lib/costModel.ts, web/lib/hardware.ts

Formulas are derived from the kipp.ly transformer inference arithmetic roofline model and the JAX scaling book. Calibration data from the tinker-cookbook recipes (math_rl, code_rl, harbor_rl).

Open the calculator →