Abstract:Amos et al. (2024) showed that the accuracy of Transformer models in sequence classification can be significantly improved by first pretraining with a masked token prediction objective without external data or augmentation, a procedure referred to as self-pretraining (SPT). While the primary objective of Amos et al. (2024) was to showcase that Transformers can achieve strong performance on the Long-Range Arena (LRA), their pipeline raises more fundamental questions: How does SPT drive optimization to better solutions? Why can standard supervised training fail in Transformers? To better understand this, we replicate and systematically ablate the findings of Amos et al. (2024). Our ablations suggest that a central bottleneck in the studied settings is not depth or generalization alone, but the ability of label supervision to learn useful query-key Attention patterns from random initialization. With a minimal setup, we identify learning proximity interactions - turning absolute positional encodings into proximity-biased Attention scores - as a key source of the improvements brought by SPT. Finally, in a simplified theoretical setup, we show that label supervision can be locally blind to certain Attention-score directions that are instead detectable through masked reconstruction.
| Comments: | v1: Preliminary, extension of the version accepted at ICML 2025 Workshop MOSS |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.21070 [cs.LG] |
| (or arXiv:2605.21070v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.21070 arXiv-issued DOI via DataCite (pending registration) |
From: Antonio Orvieto [view email]
[v1]
Wed, 20 May 2026 11:56:15 UTC (4,468 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。