























Abstract:Grokking is the phenomenon whereby, unlike the training performance, which peaks early in the training process, the test/generalization performance of a model stagnates over arbitrarily many epochs and then suddenly jumps to usually close to perfect levels. In practice, it is desirable to reduce the length of such plateaus, that is to make the learning process "grok" faster. In this work, we provide new insights into grokking. First, we show both empirically and theoretically that grokking can be induced by asymmetric speeds of (stochastic) gradient descent, along different principal (i.e singular directions) of the gradients. We then propose a simple modification that normalizes the gradients so that dynamics along all the principal directions evolves at exactly the same speed. Then, we establish that this modified method, which we call egalitarian gradient descent (EGD) and can be seen as a carefully modified form of natural gradient descent, groks much faster. In fact, in some cases the stagnation is completely removed. Finally, we empirically show that on classical arithmetic problems such as modular addition and sparse parity problem which this stagnation has been widely observed and intensively studied, that our proposed method eliminates the plateaus.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2510.04930 [cs.LG] |
| (or arXiv:2510.04930v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2510.04930 arXiv-issued DOI via DataCite |
From: Ali Saheb Pasand [view email]
[v1]
Mon, 6 Oct 2025 15:40:36 UTC (158 KB)
[v2]
Thu, 14 May 2026 19:54:00 UTC (1,860 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。