Mixture-of-Experts models are memory hogs: Qwen3.5-122B keeps 256 experts in each of 48 layers, and 95% of its roughly 250 GB of parameters sits in expert MLPs of which only 8 per layer are ever active for a given token. The direct fix is expert pruning — permanently delete the least useful experts, no retraining, composable with quantization. The hard part is deciding which ones.
Why first-order methods fail. Existing approaches score each expert independently and cut per layer. But experts are not independent: they form cooperative groups that get co-activated together, a pattern that emerges early in training and stabilises. Deleting an expert that looks minor on its own can dismantle a whole cluster. HOPE (Higher-Order Pruning of Experts) makes this explicit: it defines a surrogate that strictly upper-bounds the dominant component of pruning error and, because it is a squared sum, expanding it produces the pairwise interaction terms that first-order methods throw away. The optimal prune-set then falls out of a quadratic program over an interaction matrix F, solved via continuous relaxation.
The relationship to REAP. The diagonal of F turns out to recover REAP’s expert rankings almost perfectly (Spearman ρ = 0.988) — which means HOPE is REAP plus cross-terms, and any gain over REAP is attributable purely to the second-order structure.
Results. Across Qwen3.5-122B, Qwen3.5-35B and GLM-4.5-Air, two calibration sets (Evol-CodeAlpaca, SWE-Bench Verified trajectories) and the Tulu3 suite plus SWE-Bench Pro, HOPE takes the best average rank of five methods: 2.07 overall, and 1.58 at 50% pruning versus 2.42 for the next-best. The headline: on Qwen3.5-35B-A3B at 50% pruning HOPE scored 42.3 on SWE-Bench Pro — matching the unpruned model’s 41.8 — while REAP fell to 36.2, a 13% regression. Gains peak exactly where diverse expert combinations get exercised over long agentic trajectories.
Practical cost is a single calibration forward pass plus a non-convex QP solved in 1–2 seconds per layer. AWS Labs ships a package with the three steps calibrate → solve → prune, tested on Qwen3, Qwen3-Next, Qwen3.5 and GLM-4.5.
Blog: Higher-Order Pruning of Experts · Paper: arXiv:2609.18916