Abstract:Test-time scaling improves LLM performance by generating multiple candidate solutions, yet token-level sampling requires temperature tuning that trades off diversity against stability. Fine-grained MoE, featuring hundreds of well-trained experts per layer and multi-expert activation per token, offers an unexplored alternative through its rich routing space. We empirically characterize fine-grained MoE routing and uncover an informative pattern: router scores exhibit a certain head of high-confidence experts followed by an uncertain tail of low-confidence candidates. While single-run greedy accuracy remains stable when fewer experts are activated, multi-sample pass@n degrades significantly-suggesting that the certain head governs core reasoning capability while the uncertain tail correlates with reasoning diversity. Motivated by these findings, we propose Expert-Sample, a training-free method that preserves high-confidence selections while injecting controlled stochasticity into the uncertain tail, enabling diverse generation without destabilizing outputs. Evaluated on multiple fine-grained MoE models across math, knowledge reasoning, and code tasks, Expert-Sample consistently improves pass@n and verification-based accuracy. On Qwen3-30B-A3B-Instruct evaluated on GPQA-Diamond with 32 parallel samples, pass@32 rises from 85.4% to 91.9%, and accuracy improves from 59.1% to 62.6% with Best-of-N verification.
| Comments: | 25 pages, 13 figures |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2602.02443 [cs.LG] |
| (or arXiv:2602.02443v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2602.02443 arXiv-issued DOI via DataCite |
|
| Journal reference: | International Conference on Machine Learning (ICML), 2026 |
From: Yuanteng Chen [view email]
[v1]
Mon, 2 Feb 2026 18:39:33 UTC (900 KB)
[v2]
Fri, 1 May 2026 11:10:35 UTC (909 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。