Abstract:Mixture-of-Experts (MoE) models offer dynamic computation, but are typically deployed as static full-capacity models, missing opportunities for deployment-specific specialization. We introduce PreMoE, a training-free framework that proactively compiles sparse MoE variants for targeted deployment scenarios. At its core is Predicted Expert Utility (PEU), a robust metric for estimating expert importance from router logits through high-confidence threshold filtering and logit transformation, which together stabilize utility estimation under aggressive sparsity. Using PEU scores computed on a small calibration set, PreMoE produces domain-aware expert rankings that can be used to compile either domain-specific specialists or high-efficiency multi-domain generalists, without any retraining. Across MoE models ranging from 30B to 718B parameters, PreMoE achieves up to 50\% sparsity with nearly no performance loss. It further exposes a practical deployment trade-off: specialists maximize in-domain efficiency, while synthesized generalists retain broader cross-domain capability at the same sparsity budget.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2505.17639 [cs.LG] |
| (or arXiv:2505.17639v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2505.17639 arXiv-issued DOI via DataCite |
From: Zehua Pei [view email]
[v1]
Fri, 23 May 2025 08:59:16 UTC (1,148 KB)
[v2]
Tue, 31 Mar 2026 03:57:29 UTC (6,113 KB)
[v3]
Fri, 24 Apr 2026 08:03:55 UTC (6,113 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。