Abstract:Mixture-of-Experts (MoE) models achieve remarkable performance by sparsely activating specialized experts, yet their massive parameters in experts pose significant challenges for deployment. While low-rank quantization offers a promising route to compress MoE models, existing methods still incur nonnegligible memory overhead and inference latency. To address these limitations, we propose \textsc{TileQ}, a fine-tuning-free post-training quantization (PTQ) method that employs 2D-tiling structured low-rank quantization to share low-rank factors across both input and output dimensions of MoE experts. Furthermore, we introduce an efficient inference technique for \textsc{TileQ} that fuses multiple low-rank expert computations into a single-pass operation, significantly improving hardware utilization. Experiments show that \textsc{TileQ} cuts down additional memory usage up to 10$\times$ and reduces inference latency to $\sim$5\% while preserving state-of-the-art accuracy.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.09281 [cs.LG] |
| (or arXiv:2605.09281v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.09281 arXiv-issued DOI via DataCite (pending registration) |
From: Hongyaoxing Gu [view email]
[v1]
Sun, 10 May 2026 03:10:20 UTC (1,972 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。