Abstract:Applying Transformers to irregular time-series typically requires specializations to their baseline architecture, which can result in additional computational overhead and increased method complexity. We present the Rotary Masked Autoencoder (RoMAE), which utilizes the popular Rotary Positional Embedding (RoPE) method for continuous positions. RoMAE is an extension to the Masked Autoencoder (MAE) that enables interpolation and representation learning with multidimensional continuous positional information while avoiding any time-series-specific architectural specializations. We showcase RoMAE's performance on a variety of modalities including irregular and multivariate time-series, images, and audio, demonstrating that RoMAE surpasses specialized time-series architectures on difficult datasets such as the DESC ELAsTiCC Challenge while maintaining MAE's usual performance across other modalities. In addition, we investigate RoMAE's ability to reconstruct the embedded continuous positions, demonstrating that including learned embeddings in the input sequence breaks RoPE's relative position property.
| Comments: | NeurIPS 2025 Final Camera Ready |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2505.20535 [cs.LG] |
| (or arXiv:2505.20535v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2505.20535 arXiv-issued DOI via DataCite |
|
| Journal reference: | Advances in Neural Information Processing Systems 38, NeurIPS 2025, Pages 133952-133987 |
From: Uros Zivanovic [view email]
[v1]
Mon, 26 May 2025 21:45:18 UTC (855 KB)
[v2]
Sat, 8 Nov 2025 01:53:01 UTC (1,257 KB)
[v3]
Tue, 12 May 2026 14:41:41 UTC (1,257 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。