Abstract:Mixture-of-Experts (MoE) models face memory and PCIe latency bottlenecks when deployed on commodity hardware. Offloading expert weights to CPU memory results in PCIe transfer latency that exceeds GPU computation by several folds. We present PreScope, a prediction-driven expert scheduling system that addresses three key challenges: inaccurate activation prediction, PCIe bandwidth competition, and cross-device scheduling complexity. Our solution includes: 1) Learnable Layer-Aware Predictor (LLaPor) that captures layer-specific expert activation patterns; 2) Prefetch-Aware Cross-Layer Scheduling (PreSched) that generates globally optimal plans balancing prefetching costs and loading overhead; 3) Asynchronous I/O Optimizer (AsyncIO) that decouples I/O from computation, eliminating waiting bubbles. PreScope achieves 141% higher throughput and 74.6% lower latency than state-of-the-art solutions.
| Comments: | publishing in ICS 2026 |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2509.23638 [cs.LG] |
| (or arXiv:2509.23638v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2509.23638 arXiv-issued DOI via DataCite |
|
| Related DOI: | https://doi.org/10.1145/3797905.3807834
DOI(s) linking to related resources |
From: Enda Yu [view email]
[v1]
Sun, 28 Sep 2025 04:35:12 UTC (6,110 KB)
[v2]
Thu, 16 Apr 2026 07:53:37 UTC (5,420 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。