Authors:Yuhao Qing, Guichao Zhu, Lintian Lei, Fanxin Li, Shixiong Zhao, Zekai Sun, Xiuxian Guan, Xusheng Chen, Dong Huang, Ping Luo, Yiming Qiu, Heming Cui
Abstract:Mixture-of-Experts (MoE) scales large language models cost-effectively, but expert-parallel training suffers severe straggler effects from skewed expert loads. Current systems frequently rearrange expert placement to mitigate stragglers, inflating memory footprint and migration overhead, potentially negating the benefits of load balancing. We present Fully Sharded Sparse Data Parallelism (FSSDP), a sparse-native MoE training approach that enables in-situ load balancing on every training iteration, overlapping the balancing with computation and eliminating explicit expert rearrangement together with its migration traffic and memory reserves. FSSDP keeps MoE layers sharded and sparsely materializes an ephemeral, load-balancing placement each iteration, with re-materialization to reuse available memory across layers. FSSDP is complemented by heterogeneous sharding to shift memory imbalance from the device level to the layer level, maintaining uniform memory budgets while enabling per-layer placement optimization. We realize FSSDP in Themis with co-designed topology-aware placement algorithms. Across 2 clusters and diverse workloads, Themis achieves 1.26-2.42x speedup over state-of-the-art expert-rearrangement systems.
From: Yuhao Qing [view email]
[v1]
Tue, 4 Feb 2025 18:56:00 UTC (803 KB)
[v2]
Fri, 21 Aug 2026 11:05:29 UTC (922 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。