Abstract:Modern deep learning science often assumes that neural networks learn from a fixed data distribution. However, many practically important learning problems involve data distributions that change throughout training. How does such non-stationarity impact the inductive biases of deep learning towards models with different structural, generalisation, and safety properties? A fruitful testbed for studying inductive bias is in-context linear regression sequence modelling, where small transformers display strikingly different generalisation patterns depending on the diversity of the (fixed) training task distribution. In this paper, we explore the effect of diversifying the task distribution across training time, finding that such temporal diversity leads to an increased bias towards generalisation over memorisation.
| Comments: | Presented at Technical AI Safety Conference (TAIS), Oxford, May 2026. Code available at this https URL |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.18281 [cs.LG] |
| (or arXiv:2605.18281v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.18281 arXiv-issued DOI via DataCite (pending registration) |
From: Matthew Farrugia-Roberts [view email]
[v1]
Mon, 18 May 2026 12:12:16 UTC (10,667 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。