




















Abstract:A unified framework for first-order optimization algorithms fornonconvex unconstrained optimization is proposed that uses adaptivelypreconditioned gradients and includes popular methods such as full anddiagonal AdaGrad, AdaNorm, as well as adpative variants of Shampoo andMuon. This framework also allows combining heterogeneous geometriesacross different groups of variables while preserving a unifiedconvergence analysis. A fully stochastic global rate-of-convergenceanalysis is conducted for all methods in the framework, with andwithout two types of momentum, using reasonable assumptions on thevariance of the gradient oracle and without assuming boundedstochastic gradients or small enough stepsize.
| Subjects: | Machine Learning (cs.LG) |
| MSC classes: | 90C26, 90C06, 90C15, 90C30 |
| ACM classes: | I.2.6; F.2.1; G.1.6 |
| Cite as: | arXiv:2604.17423 [cs.LG] |
| (or arXiv:2604.17423v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2604.17423 arXiv-issued DOI via DataCite |
From: Philippe Toint [view email]
[v1]
Sun, 19 Apr 2026 13:07:51 UTC (30 KB)
[v2]
Fri, 1 May 2026 14:46:49 UTC (30 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。