























Abstract:Complex-valued Transformers have largely inherited softmax attention from real-valued architectures. However, row-normalised token competition is not necessarily aligned with phase-preserving computation. In this paper, we introduce the Phase-Coherent Transformer (PCT), which applies a real-valued, element-independent, smooth gate to L2-normalised complex query-key similarities. PCT replaces token competition with token-non-competing attention and is designed to preserve phase information across layers.
Across mid-scale benchmarks spanning long-range memory, hierarchical long-range reasoning, positional retrieval, phase-based memory and superposition, and image classification, PCT shows strong generalisation across task categories. Under parameter-fair comparison, PCT consistently outperforms both the standard softmax Transformer and its direct complex-valued counterpart. Moreover, even on tasks traditionally considered difficult for complex-valued neural networks, such as NIAH and LRA-Text, PCT remains competitive with Multiscreen, the strongest real-valued NN baseline in our comparison. Experiments introducing gates that deliberately violate the PCT conditions show that the design is not incidental: smooth gates that preserve negatively aligned phase components remain strong, whereas gates that delete such components collapse on long-range retrieval, and gates whose outputs become excessively large suffer clear performance degradation. PCT also shows no depth-related accuracy collapse across the tested depth range. These results support introducing multi-layer phase-coherent structure into attention as a promising design principle for achieving generalisation in complex-valued Transformers.
| Comments: | 26 pages, 17 tables (no figures). Companion Lean 4 formalization of Theorems 1 and 2 at this https URL |
| Subjects: | Machine Learning (cs.LG) |
| MSC classes: | 68T07, 68V20, 60J10 |
| ACM classes: | I.2.6; I.5.1; I.2.7 |
| Cite as: | arXiv:2605.10123 [cs.LG] |
| (or arXiv:2605.10123v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.10123 arXiv-issued DOI via DataCite (pending registration) |
From: Leona Hioki [view email]
[v1]
Mon, 11 May 2026 07:38:52 UTC (31 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。