LoRA & MoE
LoRA saves FLOPs and memory. MoE gives you more capacity at the same cost.
LoRA: massive FLOP and memory savings
Tinker exclusively uses LoRA adapters rather than full fine-tuning. Their research paper "LoRA Without Regret" makes the case: RL requires roughly 1,000x less information per token than supervised learning, meaning LoRA with rank ≥32 matches full fine-tuning for RLVR.
Caveat: This result holds well for math and reasoning tasks with clean reward signals. For other domains, LoRA can perform significantly worse than full fine-tuning depending on the task and even the random seed. The gap tends to widen for tasks requiring broader weight updates (e.g., multilingual, long-form generation, or tasks far from the pretrained distribution). If you're working outside the well-validated RL-on-math regime, benchmark LoRA against full FT on your specific task before committing.
LoRA FLOP math (from "LoRA Without Regret")
Full model: 3N² multiply-adds per weight matrix (fwd + bwd)
LoRA: 2N² + 6NR multiply-adds (base fwd + bwd on A,B matrices)
For R ≪ N, that's ~2/3 of full fine-tuning FLOPs
VRAM: GRPO + LoRA vs GRPO + Full Fine-Tune
GRPO + LoRA (8B model)
32.5 GB total — needs 2x H100 (160GB)
GRPO + Full FT (8B model)
128.0 GB total — needs 4x H100 (320GB)
Bars show fixed memory (model weights + optimizer states) only. Remaining GPU memory is used by vLLM's KV cache during rollout and activations during training — these two phases timeshare the same memory, so we take the max, not the sum. vLLM's gpu_memory_utilization controls how much free memory it claims for KV cache; more KV cache = more concurrent rollout sequences = faster steps. With LoRA, you have ~128 GB free across 2 GPUs for KV cache. With full FT, only ~192 GB free across 4 GPUs — and those 4 GPUs cost 2x more to rent.
But the real payoff goes beyond single-user FLOP savings. Since base weights are frozen and adapters are tiny (10-100MB, sub-second swap), LoRA unlocks multi-tenant GPU sharing — multiple users training different adapters on the same base model. This is how managed platforms like Tinker keep GPUs busy even when individual jobs have idle phases. See Tinker Multiplexing for how this works in practice.
MoE: more capacity at the same compute cost
Mixture-of-Experts models give you the representational capacity of a large model while only paying for a fraction of the parameters per token. The compute cost scales with active parameters (experts routed to per token), not the total parameter count. This creates a surprising pricing inversion:
In fact, the capacity advantage compounds during RL training: ScaleRL found that an MoE model (17B×16 experts) reached a higher performance ceiling than a dense 8B model while using only 1/6 of the RL compute. More total parameters means more latent capability for RL to surface — the base model already “knows” more, so each training step extracts more signal. See Convergence & Scale RL for the full sigmoid comparison.
| Model | Total | Active | Sample $/M | Train $/M |
|---|---|---|---|---|
| Qwen3-8B (dense) | 8B | 8B | $0.40 | $0.40 |
| Qwen3-30B-A3B (MoE) | 30.5B | 3.3B | $0.30 | $0.36 |
| Qwen3-32B (dense) | 32B | 32B | $1.47 | $1.47 |
| DeepSeek-V3.1 (MoE) | 671B | 37B | $2.81 | $3.38 |
The Qwen3-30B-A3B at $0.30/M sample tokens costs less than the dense Qwen3-8B at $0.40/M — despite being a 30B-parameter model. This is the MoE advantage: you get the representational capacity of 30B parameters but only pay for the 3.3B that are active per token.
The hidden cost of MoE
MoE flips the dominant cost from compute (active params x tokens) to memory (total params x VRAM for loading). DeepSeek V3's 671B total weights need massive VRAM just to load all experts, even though only 37B are active. Cross-node expert parallelism requires expensive all-to-all communication.
On Tinker, this complexity is abstracted away — you just see the per-token price. For self-hosted, MoE VRAM requirements can mean needing 2-4x more GPUs than a dense model with similar active params.
LoRA on MoE: shared outer dimensions
A natural question: if LoRA adapters are applied to all weight matrices, does an MoE model with 64 experts get 64x more LoRA parameters than a dense model? No — Tinker uses a clever optimization called shared outer LoRAs.
For each expert's FFN projection (gate_proj, up_proj, down_proj), the LoRA adapter has two dimensions: the outer dimension (d_model, the hidden size shared across the whole model) and the intermediate dimension (expert-specific FFN width). Shared outer LoRAs count the d_model dimension once across all experts, while the intermediate dimension is counted per-expert. This means the expert LoRA cost scales with the number of experts times their intermediate dim, not with the full expert weight size.
Confirmed in the tinker-cookbook: get_lora_param_count() iterates over all weight matrices. For expert weights, the outer dim is added only for expert_idx == 0 (shared once), while the intermediate dim is added for every expert. Controlled by shared_expert_outer_loras=True (default).
The A matrix (d_model → rank) is shared across all experts. Each expert gets its own B matrix (rank → intermediate). Total LoRA params = rank × (d_model + N × intermediate_dim), not rank × N × (d_model + intermediate_dim).
Two things are not trained with LoRA on MoE models:
- Router/gate modules — the tiny networks that decide which experts activate for each token are frozen. Routing decisions stay fixed during fine-tuning. This is standard practice: you want the model to use its pretrained routing knowledge, not learn new routing patterns from limited RL data.
- Embeddings and already low-rank projections — e.g. DeepSeek V3's MLA uses compressed q_b_proj/kv_b_proj that are already low-rank by design.
The Tinker API exposes train_mlp and train_attn toggles so you can control which modules get LoRA adapters. You can compute exact LoRA param counts with tinker_cookbook.hyperparam_utils.get_lora_param_count(model, detailed=True) which breaks down expert vs non-expert parameters.