






















Abstract:We introduce SODA, a generalization of Optimistic Dual Averaging, which provides a common perspective on state-of-the-art optimizers like Muon, Lion, AdEMAMix and NAdam, showing that they can all be viewed as optimistic instances of this framework. Based on this framing, we propose a practical SODA wrapper for any base optimizer that eliminates weight decay tuning through a theoretically-grounded $1/k$ decay schedule. Empirical results across various scales and training horizons show that SODA consistently improves performance without any additional hyperparameter tuning.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.11172 [cs.LG] |
| (or arXiv:2605.11172v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.11172 arXiv-issued DOI via DataCite (pending registration) |
From: Thomas Pethick [view email]
[v1]
Mon, 11 May 2026 19:30:47 UTC (782 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。