Abstract:Spectral bias, the tendency of neural networks to learn low frequencies first, can be both a blessing and a curse. While it enhances the generalization capabilities by suppressing high-frequency noise, it can be a limitation in scientific tasks that require capturing fine-scale structures. The delayed generalization phenomenon known as grokking is another barrier to rapid training of neural networks. Grokking has been hypothesized to arise as learning transitions from the NTK to the feature-rich regime. This paper explores the impact of preconditioned gradient descent (PGD), such as Gauss-Newton, on spectral bias and grokking phenomena. We demonstrate through theoretical and empirical results how PGD can mitigate issues associated with spectral bias. Additionally, building on the rich learning regime grokking hypothesis, we study how PGD can be used to reduce delays associated with grokking. Our conjecture is that PGD, without the impediment of spectral bias, enables uniform exploration of the parameter space in the NTK regime. Our experimental results confirm this prediction, providing strong evidence that grokking represents a transitional behavior between the lazy regime characterized by the NTK and the rich regime. These findings deepen our understanding of the interplay between optimization dynamics, spectral bias, and the phases of neural network learning.
| Comments: | 21 pages, 13 figures, |
| Subjects: | Machine Learning (cs.LG) |
| MSC classes: | 68T07, 65K10 |
| ACM classes: | I.2.6; G.1.6 |
| Report number: | SAND2026-15617O |
| Cite as: | arXiv:2601.03162 [cs.LG] |
| (or arXiv:2601.03162v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2601.03162 arXiv-issued DOI via DataCite |
From: Shuai Jiang [view email]
[v1]
Tue, 6 Jan 2026 16:35:07 UTC (4,533 KB)
[v2]
Thu, 7 May 2026 04:15:05 UTC (4,536 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。