tinker-nomics

Algorithms

RL algorithms and their cost/memory profiles.

GRPO

Group Relative Policy Optimization

DeepSeekMath (2024)

Models in memory

policy, reference

Needs critic

No

Needs reward model

No

Memory profile

low

Typical K

1

Typical G

2, 8, 16, 32, 64

Async support

Yes

ScaleRL sigmoid params (Llama-3.1-8B, MATH) — see convergence chart

A 50%
B 1.8
Cmid 4,000 GB200-hrs
≈ 11,200 H100-eq hrs

Eliminates critic/value model. ~50% memory savings over PPO. G is literal: G=16 means 16 rollout completions per prompt. 2-GRPO is an optional algorithm variant (see key_references.2_grpo) that claims G=2 matches G=16 — treat as a potential optimization, not a default assumption. ScaleRL scaling: A=50%, C_mid=4000 GB200-hrs (~11200 H100-eq), lowest ceiling of tested algorithms.

PPO

Proximal Policy Optimization

Schulman et al. (2017)

Models in memory

policy, reference, critic, reward_model

Needs critic

Yes

Needs reward model

Yes

Memory profile

high

Typical K

2, 3, 4

Typical G

1, 4, 8

Async support

Yes

4-8x more VRAM than GRPO+LoRA. Standard for RLHF.

REINFORCE

REINFORCE with baseline

Williams (1992)

Models in memory

policy, reference

Needs critic

No

Needs reward model

No

Memory profile

low

Typical K

1

Typical G

8, 16, 32

Async support

Yes

Tinker uses importance_sampling loss fn. Simpler than GRPO.

DPO

Direct Preference Optimization

Rafailov et al. (2023)

Models in memory

policy, reference

Needs critic

No

Needs reward model

No

Memory profile

low

Typical K

1

Typical G

N/A

Async support

No

Offline, no rollout generation. Uses preference pairs. Not RLVR.

DAPO

Dynamic Asymmetric Policy Optimization

Yu et al. (ByteDance, 2025)

Models in memory

policy, reference

Needs critic

No

Needs reward model

No

Memory profile

low

Typical K

1

Typical G

8, 16, 64

Async support

Yes

ScaleRL sigmoid params (Llama-3.1-8B, MATH) — see convergence chart

A 52%
B 2
Cmid 3,500 GB200-hrs
≈ 9,800 H100-eq hrs

GRPO variant with asymmetric clipping (eps_low=0.2, eps_high=0.28). Clip-Higher prevents entropy collapse. Dynamic sampling removes zero-advantage samples. 50% fewer steps than R1-Zero on AIME-24. ScaleRL scaling: A=52%, C_mid=3500 GB200-hrs (~9800 H100-eq).

Dr_GRPO

Dr. GRPO (Debiased GRPO)

Liu et al. (2025)

Models in memory

policy, reference

Needs critic

No

Needs reward model

No

Memory profile

low

Typical K

1

Typical G

8, 16

Async support

Yes

ScaleRL sigmoid params (Llama-3.1-8B, MATH) — see convergence chart

A 51%
B 1.9
Cmid 3,800 GB200-hrs
≈ 10,640 H100-eq hrs

Removes length and difficulty biases from GRPO by normalizing advantages. Unbiased gradients. 43.3% AIME-24 with 7B in ~27h on 8xA100. ScaleRL scaling: A=51%, C_mid=3800 GB200-hrs (~10640 H100-eq).

CISPO

Clipped Importance Sampling Policy Optimization

Chen et al. (2024); Khatri et al. (2024)

Models in memory

policy, reference

Needs critic

No

Needs reward model

No

Memory profile

low

Typical K

1

Typical G

8, 16

Async support

Yes

ScaleRL sigmoid params (Llama-3.1-8B, MATH) — see convergence chart

A 58%
B 2.2
Cmid 3,000 GB200-hrs
≈ 8,400 H100-eq hrs

Uses clipped importance ratio as a coefficient on the policy gradient — distinct from PPO which clips the objective directly. ~2x faster convergence than DAPO. ScaleRL scaling: A=58%, C_mid=3000 GB200-hrs (~8400 H100-eq). arXiv:2506.13585 (Chen et al.) + arXiv:2510.13786 (Khatri et al.). Available as Tinker built-in loss function.

TTT

Test-Time Training (RL)

Yuksekgonul et al. (2026)

Models in memory

policy, reference

Needs critic

No

Needs reward model

No

Memory profile

low

Typical K

1

Typical G

32, 64

Async support

Yes

RL is run at inference time on a specific problem instance rather than offline. The model adapts its weights during the solve. TTT-Discover (2026) used G=64, ran on Tinker with gpt-oss-120b, achieved SOTA on math, GPU kernels, algorithms, and biology at ~few hundred dollars per problem. Distinct from standard post-training: each run targets one problem, not a distribution.

DRO

Direct Reward Optimization

Richemond et al. (2024); Kimi Team (2025)

Models in memory

policy, reference

Needs critic

No

Needs reward model

No

Memory profile

low

Typical K

1

Typical G

8, 16

Async support

Yes

General off-policy (and offline) RL method using a quadratic penalty to constrain the policy update. Beta parameter controls the penalty strength. arXiv:2405.19107 (Richemond et al.) + arXiv:2501.12599 (Kimi Team, Kimi k1.5). Available as Tinker built-in loss function.

OPD

On-Policy Distillation

Agarwal et al. (2023); Thinking Machines Lab (2025)

Models in memory

policy, teacher

Needs critic

No

Needs reward model

No

Memory profile

low

Typical K

1

Typical G

4

Async support

Yes

Student samples from itself (on-policy); teacher provides per-token reverse KL as dense reward signal — no scalar reward needed. ~50–100x faster than RL because dense per-token gradients replace sparse outcome rewards. Works for self-distillation (student = teacher). Typical batch=64, G=4. Original algorithm: Agarwal et al. arXiv:2306.13649 ('On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes'). Tinker implementation: thinkingmachines.ai/blog/on-policy-distillation/. Available as Tinker built-in.

SDPO

Self-Distillation Policy Optimization

Gulcehre et al. (2025)

Models in memory

policy, reference

Needs critic

No

Needs reward model

No

Memory profile

low

Typical K

1

Typical G

4, 8

Async support

Yes

Uses textual feedback (errors, comments) as dense signals. Treats the current model conditioned on its own feedback as a self-teacher; computes KL divergence between student and feedback-conditioned teacher as the training loss. Improves sample efficiency over standard RL across reasoning, tool use, and code tasks. arXiv:2601.20802.

FLOP Formulas

ComponentFormula
rollout per token2 * P_active
ref logprobs per token2 * P_active
training per token6 * P_active
lora training per token2 * P_active + 6 * P_active * R / D (approx 2/3 of full FT for R << D)

P_active = active params, R = LoRA rank, D = d_model

VRAM Formulas

GRPO LoRA

P_total * 2 (base bf16) + P_ref * 2 (reference) + ~500MB (LoRA r=32 + optimizer) + KV_cache + activations

PPO full FT

P_total * 16 (weights+optim+grads) + P_critic * 16 + P_ref * 2 + P_RM * 2

Key References

cispo_chenCISPO: Clipped Importance Sampling Policy OptimizationChen et al. — arXiv 2506.13585
cispo_khatriCISPO (independent concurrent work)Khatri et al. — arXiv 2510.13786
dro_richemondDirect Reward OptimizationRichemond et al. — arXiv 2405.19107
dro_kimiKimi k1.5: Scaling Reinforcement Learning with LLMsKimi Team — arXiv 2501.12599
lora_without_regretLoRA Without RegretThinking Machines Lab
2_grpoIt Takes Two: Your GRPO Is Secretly DPOarXiv 2510.00977
laminarLaminar: async RL frameworkarXiv 2510.12633
arealAReaL: async RLarXiv 2505.24298
rlhfuseRLHFuseNSDI'25
rollartROLLART: disaggregated rollout/trainingarXiv 2512.22560
punicaPunica: SGMV multi-LoRA servingMLSys'24
s_loraS-LoRA: thousands of concurrent adaptersarXiv 2311.03285
qerlQeRL: NVFP4 quantization + LoRA2025
deepseek_v3DeepSeek V3 Technical ReportarXiv 2412.19437
dualpipeDualPipe: bidirectional pipeline parallelismDeepSeek
epoch_r1What went into training DeepSeek-R1?Epoch AI
tinker_docsTinker DocumentationThinking Machines Lab
tinker_cookbookTinker CookbookThinking Machines Lab
tinker_under_the_hoodTinker: Under the HoodThinking Machines Lab
ttt_discoverLearning to Discover at Test TimearXiv 2601.16175
era_of_experienceThe Era of ExperienceSilver, Singh, Precup & Sutton — DeepMind (2025)
skyrl_agentSkyRL-Agent: Efficient RL Training for Multi-turn LLM AgentarXiv 2511.16108
deepsweDeepSWE: Training a Frontier Code Agent End-to-End with RLTogether AI (2025)
prime_rlINTELLECT-2: Globally Decentralized Reinforcement LearningPrime Intellect — arXiv 2505.07291
ben_andersonAnatomy of a Finetuning APIBenjamin Anderson
pipeline_rlPipelineRL: Faster On-policy RL for Long Sequence GenerationPiché et al., ServiceNow — arXiv 2509.19128
vcpoVCPO: Variance Controlled Policy OptimizationarXiv 2602.17616
on_policy_distillation_originalOn-Policy Distillation of Language Models: Learning from Self-Generated MistakesAgarwal et al. — arXiv 2306.13649
on_policy_distillation_tinkerOn-Policy Distillation (Tinker implementation)Thinking Machines Lab
sdpoSelf-Distillation Policy OptimizationGulcehre et al. — arXiv 2601.20802
aweers_rl_for_llmsRL for LLMsAweers (2026)
rlhf_bookRLHF BookLambert (2026)

Further Reading

RL for LLMs(Aweers, 2026) — comprehensive walkthrough of GRPO, PPO, and modern RL algorithms for language models.
RLHF Book(Lambert, 2026) — in-depth treatment of alignment algorithms and their compute requirements.
Laminar framework— empirical analysis confirming rollout dominates RL training time (up to 83% for reasoning tasks).