Abstract:Transformer-based models can perform complicated reasoning by generating reasoning paths token by token. While effective, this approach often requires generating thousands of tokens to solve a single problem, which can be slow and computationally expensive. More importantly, it involves a discrete sampling operation at the end of each time step, creating an information bottleneck across time steps. In this work, we propose MarCos, an improvement of the transformer structure that allows fully continuous reasoning at the thought level. Unlike traditional transformer layers, which focus on refining token predictions at each time step, layers in MarCos map a continuous representation of a stepwise thought to the distribution of the next thought. This enables us to achieve multi-step reasoning in a single pass of MarCos. Preliminary experimental results on synthetic and real-world math tasks show the great potential of MarCos. Notably, we observe that the increased information bandwidth of MarCos elicits the ability of parallel thinking, in contrast to single-threaded thinking in traditional transformers. Meanwhile, in real-world math tasks, MarCos achieves more than $10\times$ speedup in wall-clock time with the same level of accuracy. Our code is available at this https URL.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2509.25020 [cs.LG] |
| (or arXiv:2509.25020v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2509.25020 arXiv-issued DOI via DataCite |
From: Jiayu Liu [view email]
[v1]
Mon, 29 Sep 2025 16:44:22 UTC (518 KB)
[v2]
Sat, 2 May 2026 05:12:24 UTC (552 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。