Abstract:The remarkable success of modern AI has been closely tied to scaling laws, yet the finite supply of high-quality data makes data efficiency--learning more from less--an increasingly important frontier. A model's inductive bias is a critical lever for data efficiency, but foundational sequence models such as State Space Models (SSMs) often rely on fixed, task-agnostic biases. When this fixed prior is misaligned with the underlying structure of a task, the model may require additional samples to overcome its own bias before learning the relevant signal. In this work, we introduce a principled framework for understanding and aligning the inductive bias of linear time-invariant SSMs. We first formalize this bias through an SSM-induced kernel and show theoretically and empirically that its spectrum is governed by the model's frequency response. This characterization motivates Task-Dependent Initialization (TDI), a fast power-spectrum matching method that aligns the initial SSM bias with the task's spectral characteristics before downstream training. Across controlled synthetic experiments, trainable one-layer SSMs, and deep SSMs on diverse real-world benchmarks, TDI can improve data-efficient generalization primarily when task-relevant spectral structure is present and the default SSM bias is spectrally mismatched. Our results provide both a theoretical lens and a practical tool for task-adaptive inductive bias, suggesting a path toward more data-efficient sequence modeling.
| Subjects: | Machine Learning (cs.LG) |
| MSC classes: | 68Q32 |
| ACM classes: | I.2.6 |
| Cite as: | arXiv:2509.20789 [cs.LG] |
| (or arXiv:2509.20789v4 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2509.20789 arXiv-issued DOI via DataCite |
From: Qiyu Chen [view email]
[v1]
Thu, 25 Sep 2025 06:14:44 UTC (856 KB)
[v2]
Fri, 26 Sep 2025 05:57:47 UTC (856 KB)
[v3]
Thu, 27 Nov 2025 06:46:48 UTC (1 KB) (withdrawn)
[v4]
Tue, 5 May 2026 03:15:54 UTC (2,111 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。