The simplest RLVR workload: single-turn math problem solving. The model generates short chain-of-thought completions (~512 tokens), and reward is binary exact-match against ground truth answers. No sandbox, no multi-turn, no environment interaction.

This is the bread-and-butter of RLVR research — MATH and GSM8K are the most commonly used benchmarks. The tinker-cookbook math_rl recipe achieves 0.768 MATH accuracy at ~180 steps with Qwen3-8B.

Completions grow ~1.5x over training as the model learns to produce longer chain-of-thought reasoning. The cost structure is dominated by rollout token generation (short completions but very large batch × group).

Code generation with sandbox verification: the model writes a solution, then a sandbox compiles and runs it against a hidden test suite. Reward is binary pass/fail. The cookbook uses Deepcoder tasks and CodeContests with SandboxFusion (local Docker) or Modal as the sandbox backend.

The cookbook sets max_turns=2 — the model gets one attempt plus one optional retry. Completions are long (~24K tokens for full programs with chain-of-thought reasoning). Each test gets a 6-second timeout, keeping sandbox execution fast.

Length growth is 2× as the model learns to write longer, more elaborate solutions with detailed reasoning. The SkyRL-Harbor integration validates with CodeContests as the reference benchmark.

The most complex RLVR workload: multi-turn agentic interaction with a sandboxed terminal. The model iteratively issues bash commands (max_tokens=8192 per turn), observes output, reasons, and issues more commands — up to 10 turns per trajectory with a 32K token cap.

This is the SkyRL + Harbor integration. Harbor manages sandbox lifecycle (Daytona or Modal), the agent loop, and reward verification via test.sh scripts. SkyRL handles GRPO training. Each Harbor task is a simple directory: instruction.md + Dockerfile + test.sh. Tasks span SWE, sysadmin, security, data science — anything verifiable in a terminal (TerminalBench).

The cookbook defaults to Kimi-K2-Thinking (1T total params, 32B active MoE) — a frontier model requiring 70+ H100s or 30+ B200s for inference. Sandbox timeouts are generous: 600s session, 120s per command, 60s for grading. This is where managed infrastructure shines: you pay per token, not per GPU.

Length growth is 2.5× — the highest of all workloads. Growth comes from two compounding sources: the model learns to use more turns (SkyRL-Agent: turns grew 18→25, +39%) and produces longer responses per turn. The 32K trajectory cap provides a hard ceiling. The SkyRL blog highlights "high variance in rollout times" and "stragglers that dominate batch completion" as key systems challenges.

Multi-turn agentic SWE-bench RL: the model acts as a coding agent — reading files, editing code, running tests — over 25 turns per trajectory. Based on the DeepSWE (Agentica) setup using a 32B model, matching the batch/group configuration used by both DeepSWE and SkyRL-Agent (B=64, G=8).

Unlike harbor_rl, the agent has longer conversations with substantial tool output at each turn (~750 observation tokens per turn from bash/grep/cat output). The model produces ~350 output tokens per turn. Conversation context grows quadratically — each turn prefills all prior turns.

This workload highlights the multi-turn prefill cost problem. On Tinker, prefill is billed per rollout (not per group), so the G=8 rollouts sharing a prompt each pay full prefill. With 25 turns of growing context, prefill dominates total cost. Self-hosted with KV caching pays ~8% of raw prefill compute.

No tinker-cookbook recipe exists for this workload — parameters are derived from DeepSWE (Agentica, 2025) and SkyRL-Agent (arXiv 2511.16108). DeepSWE used Qwen2.5-Coder-32B-Instruct with 512 Docker containers per RL iteration across 1,000+ CPU cores. SkyRL-Agent observed turns growing from 18→25 over training.

The most infrastructure-heavy RLVR workload: multi-step GUI interaction inside full Ubuntu desktop VMs. The agent observes screenshots (1920×1080), decides mouse/keyboard actions via a VLM, executes them, and repeats for 15-50 steps per episode. Reward is binary success/failure evaluated by 134 task-specific functions (file diffs, pixel comparison, perceptual hashing).

OSWorld (NeurIPS 2024) provides 369 tasks spanning LibreOffice, GIMP, web browsing, system administration, and cross-application workflows. Unlike terminal-based benchmarks, the agent must interpret visual state — making this inherently multimodal and much more expensive per step (screenshot encoding dominates prompt tokens).

The key tradeoff vs. simpler benchmarks: environment overhead completely dominates. Each rollout requires a full QEMU-based VM (~25 GB image), application initialization (LibreOffice alone takes ~1 min on first boot), and screenshot capture at every step. ComputerRL ran 1,024 parallel Docker containers to make training feasible, and still reported system freezes and network bottlenecks under load.

Sparse binary rewards with no intermediate signal make credit assignment across 15-50 steps extremely challenging. ComputerRL found that extended RL training causes entropy collapse — the model degenerates to repetitive actions — requiring a novel 'Entropulse' strategy alternating RL with SFT to maintain exploration. Per-step latency grows linearly with context (accumulated screenshot history), so later steps are ~3x more expensive than initial ones.