Abstract:For three decades online recurrent learning has been assumed to require propagating a Jacobian tensor through the network's dynamics at $O(n^4)$ per step. We show it doesn't. Dropping the propagation entirely ($d=0$, $O(n^2)$ memory) matches full RTRL within CI on held-out BCI cross-session drift (TOST equivalent within $\pm 3$ pp at $n=20$, Adam, float64), and across vanilla-RNN synthetic cells (sine and Lorenz under Adam and SGD) and LSTM/sine under Adam.
A decomposition $g_{RTRL} = g_{imm} + g_{past}$ explains why. On BCI, $g_{past}$ concentrates in a single direction (top-1 singular fraction 0.62-0.74 across four optimizers, vs 0.333 for $g_{imm}$), and the four-optimizer full-RTRL-vs-$d=0$ recovery gap tracks each optimizer's per-layer update-magnitude ratio $\|\Delta W_{hh}\|/\|\Delta W_{out}\|$ monotonically. A stationary (no-drift) control collapses both concentrations to ~0.6: the drift-specific signal is the differential, not $g_{past}$'s absolute rank-1 structure. The signature and the behavioral gap both collapse on LSTM, consistent with a mechanism specific to additive linear recurrence. On synthetic sine, $g_{imm}$ is redundant with $g_{past}$, which predicts the synthetic null. Full RTRL's one robust advantage is LARS (+17 to +27 pp), but $d=0$+LARS also fails to adapt independently; the gap is an optimizer$\times$method interaction, not a method-quality claim. We characterize the regime: $d=0$+Adam+float64 is robust; SGD, Adafactor, and float32 have specific fragilities documented in the paper. On the evaluated cells, the $1000\times$ memory saving at $n=1024$ ($O(n^2)$ vs $O(n^4)$) comes with no measured recovery cost.
| Comments: | 25 pages, 4 figures, 19 tables. Submitted to NeurIPS 2026 |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2603.28750 [cs.LG] |
| (or arXiv:2603.28750v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2603.28750 arXiv-issued DOI via DataCite |
From: Aur Shalev Merin [view email]
[v1]
Mon, 30 Mar 2026 17:54:55 UTC (55 KB)
[v2]
Fri, 3 Apr 2026 00:57:13 UTC (72 KB)
[v3]
Sun, 26 Apr 2026 23:25:03 UTC (119 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。