惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

The GitHub Blog
The GitHub Blog
有赞技术团队
有赞技术团队
Apple Machine Learning Research
Apple Machine Learning Research
V
V2EX
Engineering at Meta
Engineering at Meta
美团技术团队
H
Hackread – Cybersecurity News, Data Breaches, AI and More
博客园 - 司徒正美
I
InfoQ
S
SegmentFault 最新的问题
博客园 - 叶小钗
N
Netflix TechBlog - Medium
Y
Y Combinator Blog
IT之家
IT之家
博客园 - Franky
大猫的无限游戏
大猫的无限游戏
人人都是产品经理
人人都是产品经理
T
The Blog of Author Tim Ferriss
月光博客
月光博客
The Cloudflare Blog
U
Unit 42
GbyAI
GbyAI
L
LangChain Blog
Microsoft Azure Blog
Microsoft Azure Blog

math.PR updates on arXiv.org

Visibility in the Boolean Model on Harmonic Manifolds Global estimates on the Brenier map Geodesics and Wandering Exponents in Brochette First-Passage Percolation State-dependent inverse-subordinator time changes of regenerative processes: Excursion structure and multiscale occupation-time limits Randomly twisted transfer operators and singular values statistics Generalized Bessel-Dunkl diffusions An almost sure invariance principle for the Takagi-van der Waerden class functions Central limit theorems for high dimensional lattice polytopes: cosmological polytopes Convergence rate estimates for semigroups and heat kernels associated with resistance forms Second-order Poincaré inequalities and localization on the Poisson space Maximum Probability of Independence in Transitive Matroids On global solutions to the semidiscrete stochastic heat equation The Poisson Tail Conjecture for primes in short intervals A Complete Spectral Analysis of the CEV Operator with Applications to Arbitrage Holographic functions and neural networks From Betting to Empirical Bernstein LIL Concentration of General Stochastic Approximation Under Heavy-Tailed Markovian Noise Pointwise Generalization in Deep Neural Networks Bayesian Latent Space Models for Graphs Are Misspecified: Toward Robust Inference via Generalized Posteriors Wasserstein bounds for denoising diffusion probabilistic models via the Föllmer process A note on connections between the Föllmer process and the denoising diffusion probabilistic model Simple Approximation and Derivative Free Inference-Time Scaling for Diffusion Models via Sequential Monte Carlo on Path Measures Diffusion-Based Stochastic Operator Networks for Uncertainty Quantification in Stochastic Partial Differential Equations A Fourier perspective on the learning dynamics of neural networks: from sample complexities to mechanistic insights Propagation of Chaos in Contextual Flow Maps Dimension-Uniform Discretization Analysis of Preconditioned Annealed Langevin Dynamics for Multimodal Gaussian Mixtures $α$-TCAV: A Unified Framework for Testing with Concept Activation Vectors Scaling Laws from Sequential Feature Recovery: A Solvable Hierarchical Model On the Limits of Latent Reuse in Diffusion Models State-of-art minibatches via novel DPP kernels: discretization, wavelets, and rough objectives
Feature Learning Dynamics in Infinite-Depth Neural Networks
Zihan Yao, Ruoyu Wu, Tianxiang Gao · 2025-12-24 · via math.PR updates on arXiv.org

Deep neural networks have achieved remarkable success in practice, yet a mechanistic understanding of how features evolve during training remains incomplete, especially in the large-depth limit. For ResNets under depth-$μ$P scaling, prior work treats the layer index $\ell$ as a continuous time $t_\ell = \ell/L$, yielding SDE descriptions of the training dynamics. A key unresolved issue is that backpropagation reuses each forward weight matrix $W_\ell$ through its transpose $W_\ell^\top$, creating correlations between forward features and backward gradients whose behavior and role in feature learning remain unclear. We study this reused-weight forward--backward coupling in one-layer ResNets under depth-$μ$P. Using conditional Gaussian representations, we explicitly separate the coupling terms induced by weight reuse from decoupled Gaussian fluctuations before taking any network limit. At initialization, we prove that the coupling is a finite-width effect and vanishes at rate $O(n^{-1})$, uniformly over depth. During training, however, SGD induces a nontrivial forward--backward correlation term that survives the infinite-width limit. The key depth effect is that, under depth-$μ$P scaling, this surviving term is higher order in depth and its accumulated contribution over layers becomes negligible as $L\to\infty$. This depth-induced suppression motivates Neural Feature Dynamics (NFD), a forward--backward SDE system with decoupled backward weights that retains the feature-gradient covariance structure generated during training. Under nondegeneracy assumptions, we prove that the finite-network training dynamics converge to its NFD limit with an $O(L^{-1})$ depth-discretization error, while the reused-weight coupling term has a faster $O(L^{-2})$ decay. These results provide a rigorous infinite-depth limit for the feature-learning dynamics of one-layer ResNets under depth-$μ$P.