Abstract:Using recent works of one of the authors (VSB) on collapse in generative models and two time scale dynamics in stochastic gradient descent in high dimensions, we give a system theoretic explanation of the memorization phenomenon in generative models. This relies purely on the dynamic aspects of the training phase. Specifically, we use a result of Austin [2016] to motivate a stylized model for the loss function for stochastic gradient descent (SGD) wherein the loss function has a strong dependence on some variables and weak dependence on the rest in a precise sense. This naturally leads to two distinct time scales in the constant step size SGD that is commonly used in machine learning. This fact has been used to explain the double descent phenomenon in SGD in Borkar [2026]. In conjunction with a mathematical model for collapse phenomenon in SGD developed in Borkar [2025a], we analyze the constant step size SGD using the recent results of Azizian et al. [2024] in order to explain the phenomenon of memorization wherein a generative model that is concurrently being tuned yields the same or similar outputs for significant stretches of time. This gives a novel perspective on the aforementioned phenomena reported in machine learning literature and their interrelationships, using a dynamical systems viewpoint.
| Comments: | 12 pages |
| Subjects: | Machine Learning (cs.LG) |
| MSC classes: | 68T07 |
| Cite as: | arXiv:2605.19483 [cs.LG] |
| (or arXiv:2605.19483v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.19483 arXiv-issued DOI via DataCite (pending registration) |
From: Vivek Borkar [view email]
[v1]
Tue, 19 May 2026 07:34:58 UTC (42 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。