Abstract:We report a systematic failure mode in predictive representation learning. Across 2695 neural network configurations trained to predict linear-Gaussian dynamics, the optimal encoder tracks the environment rather than the system it is meant to model. The mean causal fidelity -- the fraction of encoder sensitivity allocated to system degrees of freedom -- is 0.49, and only 2.5% of configurations exceed 0.70. The failure intensifies with dimension: at N=100, the optimal encoder becomes causally blind (fidelity ~10^{-8}) while achieving 92% lower prediction error than the causal representation. We prove this is not an optimization artifact but a structural property of the predictive objective: when environment modes are slower or less noisy than system modes, every minimizer of the population risk encodes the former. The set of dynamics exhibiting this predictive-causal gap is open and of positive measure in parameter space. In a nonlinear Duffing-GRU sweep, unconstrained predictors learn environment-dominant representations in 55% of tasks (95% CI 41--68%) versus 24% under operational grounding (p=2.3e-3); the median out-of-distribution MSE inflation under environment shift is 1.82x versus 1.00x. Operational grounding -- restricting the loss to system observables -- partially suppresses the gap, but causal fidelity is never recovered without an explicit system-environment boundary. The results identify the predictive-causal gap as a structural limit of learning, with implications for self-supervised representation learning, world models, and the scaling paradigm.
| Comments: | 15 pages, 5 figures, 3 tables. Supplemental Material included (Sections S1-S10) |
| Subjects: | Machine Learning (cs.LG) |
| MSC classes: | 68T07, 62M45, 37M10, 81S2268T07, 62M45, 37M10, 81S22 68T07, 62M45, 37M10, 81S22 |
| Cite as: | arXiv:2605.05029 [cs.LG] |
| (or arXiv:2605.05029v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.05029 arXiv-issued DOI via DataCite (pending registration) |
From: Kejun Liu [view email]
[v1]
Wed, 6 May 2026 15:25:37 UTC (138 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。