Abstract:Asynchronous stochastic gradient descent (SGD) enables scalable distributed training but suffers from gradient staleness. Existing mitigation strategies, such as delay-adaptive learning rates and staleness-aware filtering, typically attenuate or discard delayed gradients, introducing systematic bias: updates from simpler or faster-to-process samples are overrepresented, while gradients from more complex samples are delayed or suppressed. In contrast, prior approaches to data-dependent delays rely on a Lipschitz assumption that yields suboptimal rates or leave the smooth, convex case unaddressed. We propose a momentum-based asynchronous framework designed to preserve information from delayed gradients while mitigating the effects of staleness. We establish the first optimal convergence rates for data-dependent delays in both convex and non-convex smooth setups, providing a new result for asynchronous optimization under standard assumptions. Additionally, we derive robust learning-rate schedules that simplify hyperparameter tuning in practice.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.02043 [cs.LG] |
| (or arXiv:2605.02043v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.02043 arXiv-issued DOI via DataCite (pending registration) |
From: Tehila Dahan [view email]
[v1]
Sun, 3 May 2026 20:24:24 UTC (43 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。