Most of a mixture-of-experts model’s parameters live inside its experts, and standard training keeps every expert gradient resident in GPU memory until the optimizer step. That memory scales with both the number and the size of the experts — exactly the two axes MoE architectures grow along. The constraint is not compute; it is holding the bookkeeping.
Sonar-MoE, posted to arXiv on 2 November, removes the constraint by updating each expert tensor during the backward pass and freeing its gradient the moment it is no longer needed. The GPU never holds more than one layer of expert gradients at a time.
Numbers
The method is an engineering change, so the numbers are the argument. Fine-tuning OLMoE-1B-7B and Granite 3.1-3B-A800M in bf16 across two expert-kernel backends, Sonar-MoE:
- cuts peak memory by 19–22%
- takes 17–23% less time per step
- produces bitwise identical updates to standard training when no expert gradient clipping is used
- keeps held-out loss within 0.003 of every baseline
Bitwise identity is the detail that makes this more than a memory trick. The speedup and the memory saving are not bought with a change in optimization semantics — the gradients are numerically the same, so an existing training recipe reproduces exactly.
The Catch Is Batching
The obvious implementation — updating one expert at a time as its gradient arrives — costs a 20–72% slowdown at the same memory footprint. Sonar-MoE avoids that by updating whole batched tensors rather than individual experts, which keeps the fast path intact.
The result is a hybrid: it matches optimizer-in-backward over all parameters on both memory and speed, while still keeping the dense (non-expert) parameters on standard clipping. For anyone who wanted optimizer-in-backward for dense models but found it awkward to apply across a batched MoE parameter layout, this is the missing piece.
The practical upshot: MoE fine-tuning on a fixed GPU budget can now fit a larger expert configuration, and the wall-clock win is free. Both claims will need reproducing on other hardware and other expert kernels before they generalize beyond the two models and two backends tested here.
Paper: arXiv:2611.03577