
























Abstract:Even for the gradient descent (GD) method applied to neural network training, understanding its optimization dynamics, including convergence rate, iterate trajectories, function value oscillations, and especially its implicit acceleration, remains a challenging problem. We analyze nonlinear models with the logistic loss and show that the steps of GD reduce to those of generalized perceptron algorithms (Rosenblatt, 1958), providing a new perspective on the dynamics. This reduction yields significantly simpler algorithmic steps, which we analyze using classical linear algebra tools. Using these tools, we demonstrate on a minimalistic example that the nonlinearity in a two-layer model can provably yield a faster iteration complexity $\tilde{O}(\sqrt{d})$ compared to $\Omega(d)$ achieved by linear models, where $d$ is the number of features. This helps explain the optimization dynamics and the implicit acceleration phenomenon observed in neural networks. The theoretical results are supported by extensive numerical experiments. We believe that this alternative view will further advance research on the optimization of neural networks.
| Subjects: | Machine Learning (cs.LG); Numerical Analysis (math.NA); Optimization and Control (math.OC) |
| Cite as: | arXiv:2512.11587 [cs.LG] |
| (or arXiv:2512.11587v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2512.11587 arXiv-issued DOI via DataCite |
From: Alexander Tyurin [view email]
[v1]
Fri, 12 Dec 2025 14:16:35 UTC (4,242 KB)
[v2]
Thu, 21 May 2026 05:06:58 UTC (10,242 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。