Motivation
Inference economics are solved. Post-training economics are not.
The inference cost war gave us tokenomics
Over the past two years, inference pricing has collapsed. GPT-4-level intelligence went from $60/M tokens to under $1/M tokens. The competitive dynamics are well-understood: providers compete on price-per-token, users optimize for tokens consumed, and architectural innovations like GQA, MLA, and speculative decoding directly translate to cheaper tokens.
This created tokenomics — a clean mental model where cost scales transparently with usage. Everyone from indie hackers to enterprise teams can reason about inference costs.
Inference is converging — the task doesn't change the price
Inference has a clean property: cost is determined by tokens consumed, not by how hard the question is. Asking a model to solve a PhD-level math problem costs the same as asking it to say hello — assuming the same output length. Models at similar capability tiers have converged to similar price points. You shop for the cheapest model that hits your quality bar, and the economics are predictable.
This means model choice barely moves the cost needle at inference. A Qwen3-8B and a Llama-3.1-8B cost roughly the same per token. The same model handles easy and hard prompts at the same price.
We are entering the era of experience
For decades, AI systems learned from human-generated data: text scraped from the web, curated datasets, human feedback. The next era is different. Silver, Singh, Precup & Sutton (2025) argue we're entering an era where agents generate their own training data through interaction — executing code, calling tools, browsing the web, running experiments. Learning from experience, not from static human corpora.
RL is the natural training algorithm for this regime. The model acts, the environment responds, and the reward signal drives learning. But generating each training example now requires running a real environment — not scraping a web page, but running a Docker container, making API calls, executing test suites. The bottleneck shifts from data collection to RL environment curation.
This shift is fundamental. In supervised learning, the hard work is collecting and labeling data. In RL for LLMs, the hard work is building the environment: the sandbox, the reward function, the task distribution, the reset mechanism. Think of it like OpenAI Gym for language — the environment defines the MDP, and the reward function is just one component of that environment. The quality of your environment (task diversity, reward fidelity, execution reliability) bounds what RL can learn, regardless of compute budget.
Several projects are building this infrastructure layer: Harbor manages sandbox lifecycle for agentic tasks (Daytona/Modal containers with instruction + Dockerfile + test.sh), torchforge's OpenEnv provides pluggable environments with verifiable rewards and Weaver (weak verifier ensembles), and PRIME-RL's verifiers library offers composable rubric-based environments. The pattern is clear: the community is converging on environment-as-abstraction, not reward-function-as-abstraction.
Post-training economics are completely different
Unlike inference, the structure of the task drives cost in RL training — and the cost drivers are more nuanced than "hard tasks need more rollouts." Two properties of the reward landscape determine how many rollouts you need to generate a useful training signal:
Exploration hardness
How difficult is it for the policy to stumble onto a rewarding trajectory at all? If the model solves a task 5% of the time, you need roughly 20× more rollouts per gradient step to get comparable signal vs a task solved 50% of the time. You can't learn from all-zero rewards. Exploration hardness captures this: the structure of the action space, the length of the horizon, and how far from the initial policy the rewarding behaviors lie. Ladosz et al. (2022) provide a comprehensive survey of exploration methods in deep RL — the core challenge is that naive random exploration scales exponentially with horizon length, making rollout cost the dominant economic factor for tasks that require coordinated multi-step behavior.
Reward signal density
Even when the model reaches a terminal state, how informative is the reward? A binary pass/fail reward on a 50-step coding task gives one bit of signal per trajectory. A reward that provides partial credit at intermediate steps gives far more gradient signal per rollout token. Laidlaw et al. (2023) formalize this via the lookahead steps needed to distinguish good from bad actions — a reward that requires many steps of lookahead is effectively sparse, because early actions can't be easily credited. Sparse rewards mean more rollouts are needed to reduce variance in the policy gradient, directly multiplying cost. As a concrete example, OPD (Outcome-supervised Process-reward Data) demonstrates how denser reward signals at intermediate reasoning steps can dramatically improve sample efficiency — turning binary outcome rewards into step-level process rewards.
Model capability changes the economics
Both factors above interact with the starting model's capability. At inference, Qwen3-8B and a frontier model cost similarly per token. For RL training, a more capable starting model might solve 30% of problems vs 2% for a weaker one — a 15× difference in effective rollout cost per useful training sample. A stronger model effectively reduces exploration hardness (it starts closer to rewarding trajectories) and can better exploit sparse rewards (it needs fewer rollouts to see signal). Model choice is a cost lever, not just a quality lever.
Tool use multiplies token count 10–100×
A math rollout: ~512 completion tokens. A 20-turn SWE-bench trajectory: bash output (500 tokens) × 20 turns + reasoning chains = 15,000–50,000 sample tokens per trajectory. DeepSWE (Agentica, 2025) spawned 512 Docker containers per RL iteration, requiring Kubernetes across 1,000+ CPU cores just to keep pace with 64 H100s of training. SkyRL-Agent built a dedicated async dispatcher because synchronous batching couldn't keep GPUs fed during multi-turn tool-use rollouts. Longer trajectories compound both exploration hardness (more steps to coordinate) and reward sparsity (final reward must be credited across more actions).
GPU-hours hide all of this
An 8×H100 node running RLVR spends 80–90% of its time doing inference (rollout generation), but you pay training-tier rates for what is fundamentally an inference workload. During rollout, GPUs achieve single-digit MFU because autoregressive decoding is memory-bandwidth-bound. During training, they hit 30–50% MFU. The hourly bill doesn't distinguish any of this.
GQA changed inference economics. What changes post-training?
Grouped-Query Attention restructured inference economics by shrinking KV cache, enabling 4–8× more concurrent sequences, and making batch inference dramatically cheaper. The analogy for post-training is a stack of innovations that compound multiplicatively:
- LoRA multi-tenancy — shared base model, per-user adapters
- MoE architectures — pricing on active params, not total params
- Async RL — overlap rollout with training, 1.5–3.5× speedup
- Quantized rollout (FP4/FP8) — 2–4× generation acceleration
- Hardware generations — B200's 2.4× bandwidth directly speeds the bottleneck
- 2-GRPO — G=2 may match G=16 gradient quality, 8× fewer rollouts (optional; validated on math with clean rewards)
Combined, these could yield 20–50× cost reductions vs a naive H100 + synchronous veRL baseline. This guide walks through each one.