Why multiplexing matters

The economics of managed fine-tuning rest on one question: can you share GPUs across users without sacrificing quality? Full fine-tuning says no — each user needs a dedicated GPU allocation because the entire model is modified. LoRA says yes — the base model stays frozen and shared, and each user's modifications live in a tiny adapter (a few hundred MB vs tens of GB).

This isn't just a nice optimization. It's the fundamental architectural requirement for per-token pricing. If Tinker had to dedicate GPUs per user, it would charge per GPU-hour (like self-hosted). Because LoRA lets hundreds of users share the same GPU pool, Tinker can charge per token — you pay for compute consumed, not hardware reserved.

The key enabler is LoRA's radical parameter efficiency: a rank-32 adapter for an 8B model is ~198 MB versus ~68 GB for full fine-tuning (weights + optimizer + gradients). Adapter swaps are a sub-second memcpy rather than a minutes-long model reload. Full fine-tuning cannot participate in this scheme because swapping billions of parameters between users would obliterate the throughput gains.

Why LoRA's tiny footprint makes multi-tenancy physically possible

These are the same Qwen3-8B numbers from the interactive diagram above. The 82:1 ratio between base model and adapter size is what makes multi-tenancy physically possible — swapping 198 MB is a sub-second memcpy, not a minutes-long model reload.

The clock cycle: one heartbeat of multi-tenant training

Tinker assigns training jobs to a shared worker pool — GPUs executing forward-backward operations in lockstep. Each discrete step is a clock cycle. Each cycle may involve multiple LoRA models from different users being trained simultaneously.

Step time: clock cycles and batch splitting

In each clock cycle, Tinker runs forward_backward() and optim_step() together on the shared GPU pool, potentially for multiple LoRA adapters simultaneously.

We profiled Tinker's step time using the pipelined async pattern across two models and varying batch sizes. The key findings: each clock cycle has a fixed capacity per model pool, large batches get split across multiple cycles, and different models run on separate pools with different cycle times.

LoRA rank 32, 1024 tokens/completion, pipelined async, 10 steps per config (step 0 excluded as cold start). Cycle time = total wall-clock / total cycles. Source: exp_profile_step (W&B: tinker-nomics).

The cycle time is roughly constant per model but not perfectly fixed. For 4B at moderate batch (8×8), it's a tight 2.9s ± 0.2s. But at larger batch (64×16), the per-cycle time rises to ~5.2s — suggesting cycles do more work when more data is queued. At tiny batch (4×2), only 3 data points and noisier (4.0s ± 1.6s). The cycle isn't a perfectly fixed metronome; it's more like a scheduling quantum that stretches under load.

The 30B-A3B pool is noisier across the board. At small batch (4×2, warmed up), cycle time is 4.7s ± 0.6s. At larger batch (8×8), it jumps to 5.8s ± 2.7s. The first 2–3 steps are especially noisy (up to 15s) — likely cold start as the pool allocates workers. The 30B pool is probably smaller (fewer MoE users), so less statistical smoothing from other workloads. The cycle IDs are in completely different ranges (5.1M vs 3.8M), confirming separate worker pools per model.

The MoE comparison is revealing: Qwen3-30B-A3B has only 3B active params (fewer than 4B dense) but is 3.3x slower at the same batch size. Total params matter — all 30B of expert weights must be loaded even though only 3B fire per token, plus the LoRA adapter spans more linear layers.

The per-cycle variance comes from scheduling overhead, not GPU compute noise: network round-trips (your machine → Tinker API → worker pool → back), batch assembly and partitioning when splitting across cycles, adapter routing in the multi-tenant pool, and LoRA weight sync after each optimizer step. At larger batches the response payload is bigger (1M tokens of results vs 65K), which explains why per-cycle time stretches. This overhead is small relative to compute at realistic batch sizes — the important thing is that step time is stable and predictable (no drift across steps), which is what matters for cost estimation.

For cost prediction: step time ≈ ceil(B × G / capacity) × cycle_time. Profile 3–5 steps at your target batch size to get a reliable estimate, then multiply by total steps.

How the SGMV kernel works

One fused GPU kernel serving three different LoRA adapters simultaneously.

In any given clock cycle, the pool might serve a chatbot fine-tune (47 tokens), a code generation RL rollout (4,096 tokens), and a summarization job (512 tokens) — all at once. The naive approach of launching separate GPU kernels per user wastes parallelism. SGMV (Segmented Gather Matrix-Vector, from the Punica paper) fuses all users' LoRA computations into a single kernel call.

SGMV is not the attention kernel. It operates on the linear projections — the weight matrix multiplies before and after attention (q/k/v/o_proj) plus in the MLP (gate/up/down_proj). That's 7 projections per layer × 36 layers = 252 SGMV calls per forward pass. For each projection, the base model computes Y = x · W as a single shared GEMM across all users' tokens. SGMV then adds the per-user LoRA delta x · A · B.

The computation flows in three stages. First, all users' tokens are flattened into one matrix — batch and sequence dimensions collapsed, with segment indices tracking which adapter each token belongs to. Then SGMV runs in two phases: shrink (project from hidden dim 4,096 down to rank 32 via each user's A matrix) and expand (project back up via B). Each phase is one fused kernel call.

Forward pass: what actually happens

For every linear layer (q/k/v/o projections, gate/up/down MLP — 7 per layer, 252 total), LoRA adds a small correction on top of the frozen base weight. Here's the idea: each user has their own tiny adapter matrices A and B. The base model weight W is shared. The trick is doing all users at once.

for each user i:
  y_i = x_i @ W         # base matmul
  y_i += x_i @ A_i @ B_i  # LoRA fix-up

# stack all tokens, track who owns each row
x   = [user_a_tokens; user_b_tokens; user_c_tokens]
seg = [  A, A, ...,     B, B, ...,     C, C, ...   ]

y = x @ W                # one big shared matmul
y += sgmv(x, seg, As, Bs)  # one kernel, all adapters

What happens inside sgmv()? Two phases, each a single kernel launch:

def sgmv(x, seg, As, Bs):
    # Phase 1: "shrink" — project down to LoRA rank
    # each user's rows hit THEIR A matrix
    # 4096-dim  →  rank 32  (128x smaller)
    v = zeros(N, rank)
    v[seg == A] = x[seg == A] @ A_a   # user A's 47 tokens × A_a
    v[seg == B] = x[seg == B] @ A_b   # user B's 4096 tokens × A_b
    v[seg == C] = x[seg == C] @ A_c   # user C's 512 tokens × A_c
    # ↑ all happens in ONE kernel call, not three

    # Phase 2: "expand" — project back up to full dim
    # each user's rows hit THEIR B matrix
    # rank 32  →  4096-dim
    delta = zeros(N, hidden)
    delta[seg == A] = v[seg == A] @ B_a
    delta[seg == B] = v[seg == B] @ B_b
    delta[seg == C] = v[seg == C] @ B_c
    # ↑ also ONE kernel call

    return delta   # add this to the base matmul output

That's it. The base matmul x @ W dominates the compute. The LoRA part (shrink + expand) adds only 2 × rank / hidden_dim extra — for rank 32 on a 4096-dim model, that's 1.6% overhead.

Why RL needs batch-invariant kernels

On-policy RL computes logprobs for the same tokens twice — once during rollout (sampling) and once during the training forward pass. These logprobs must match exactly. If they don't, the policy gradient is computed against stale probabilities, and what should be on-policy RL silently becomes off-policy. As Horace He (Thinking Machines Lab) puts it: “the different numerics between training and inference implicitly turns our on-policy RL into off-policy RL.”

In a multi-tenant pool this is almost guaranteed to happen. The effective batch size is the total flattened token count across all users — and it changes constantly as users join, leave, or have different sequence lengths. The base model ops are where the problem lives: x @ W matmuls switch tile configurations at different batch sizes, RMSNorm flips between data-parallel and split-reduction, and attention changes its KV split count. The result: the same tokens produce bitwise-different logits depending on who else is sharing the GPU.

Notably, the SGMV adapter kernel is less affected — each user's tokens are already isolated by segment index, so the per-user reduction doesn't cross into other users' tokens. The nondeterminism danger is in the shared base model computation that happens alongside SGMV, not in the per-user adapter delta itself.

Standard kernels change their reduction strategy based on batch size — the same tokens produce different logits depending on how many other tokens are in the batch. For RL, this means rollout logprobs and training logprobs silently disagree. From Defeating Nondeterminism in LLM Inference (Horace He, Thinking Machines Lab).

The fix: make the base model ops batch-invariant — a token's output must be identical whether the flattened batch has 47 tokens or 4,655. Thinking Machines Lab's batch_invariant_ops library provides drop-in replacements for torch.mm, torch.addmm, torch.log_softmax, and torch.mean with fixed reduction orders. SGLang v0.4.6 integrated these alongside batch-invariant attention kernels (fixed split-KV size) for deterministic inference.

With batch-invariant kernels, the KL divergence between rollout and training logprobs is a flat line at exactly 0 — true on-policy RL. Without it, KL hovers around ~0.001 with intermittent spikes that correlate with reward collapse. In Thinking Machines' RLVR experiments on BigMath, the non-invariant version hit reward collapse around step 318; the batch-invariant version trained smoothly.

The tradeoff is severe — batch-invariant kernels are significantly slower (~60% overhead), which directly reduces effective FLOPs/second. In practice, it's an open question whether these are used for all workloads or only selectively for on-policy RL where logprob matching is critical. For off-policy methods, SFT, or workloads where exact reproducibility isn't required, standard (faster) kernels would be the pragmatic choice.

Both pieces are in production separately: SGMV-style multi-LoRA kernels ship in vLLM (Punica-derived, Jan 2024) and SGLang (S-LoRA, 2,000 concurrent adapters on one H100). Batch-invariant base ops ship in TML's library and SGLang's deterministic inference mode. Whether Tinker combines both in its multi-tenant training pool is not publicly confirmed — this section describes the likely architecture based on the published literature and blog posts.

The economics: why multi-tenant RL can be cheaper

Co-serving training and inference on the same GPUs is well-established in the literature. The key insight is that the two workloads have complementary resource profiles: inference is memory-bandwidth bound, training is compute bound. FlexLLM (NSDI 2026) showed 1.9–4.8x training throughput improvement and 80% memory reduction by interleaving PEFT fine-tuning with inference on shared weights. LLMStation (ATC 2025) went further with operator-level suspend/resume — pausing backward-pass nodes during latency-sensitive prefill, co-executing during decode, all with 18μs scheduler overhead.

What's new is combining these co-serving techniques with the LoRA multiplexing and batch-invariant kernels described above to build a multi-tenant RL training service that's cheaper than dedicated GPUs. The cost reduction stacks: