





















Abstract:We show that gating mechanisms in recurrent neural networks (RNNs) induce lag-dependent and direction-dependent effective learning rates, even when training uses a fixed, global step size. This behavior arises from a coupling between state-space time-scales (parametrized by the gates) and parameter-space dynamics during gradient descent. By deriving exact Jacobians for leaky-integrator and gated RNNs and applying a first-order expansion, we make explicit how constant, scalar, and multi-dimensional gates reshape gradient propagation, modulate effective step sizes, and introduce anisotropy in parameter updates. These findings reveal that gates act not only as filters of information flow, but also as data-driven preconditioners of optimization, with formal connections to learning-rate schedules, momentum, and adaptive methods such as Adam. Empirical simulations corroborate these predictions: across several sequence tasks, gates produce lag-dependent effective learning rates and concentrate gradient flow into low-dimensional subspaces, matching or exceeding the anisotropic structure induced by Adam. Notably, gating and optimizer-driven adaptivity shape complementary aspects of credit assignment: gates align state-space transport with loss-relevant directions, while optimizers rescale parameter-space updates. Overall, this work provides a unified dynamical systems perspective on how gating couples state evolution with parameter updates, clarifying why gated architectures achieve robust trainability in practice.
| Comments: | final version |
| Subjects: | Machine Learning (cs.LG); Dynamical Systems (math.DS) |
| Cite as: | arXiv:2508.12121 [cs.LG] |
| (or arXiv:2508.12121v5 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2508.12121 arXiv-issued DOI via DataCite |
From: Lorenzo Livi [view email]
[v1]
Sat, 16 Aug 2025 18:19:34 UTC (620 KB)
[v2]
Wed, 20 Aug 2025 07:10:59 UTC (1,767 KB)
[v3]
Sun, 24 Aug 2025 17:10:20 UTC (2,535 KB)
[v4]
Mon, 9 Mar 2026 10:48:51 UTC (4,883 KB)
[v5]
Tue, 21 Apr 2026 17:54:21 UTC (8,164 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。