Abstract:GPT-style language models are sensitive to single-token changes at generation points where the predicted probability distribution is spread across multiple tokens. Viewing this sensitivity as a geometric property, we derive an $\mathfrak{so}(n)$-valued 1-form that depends only on the geometry of the token embeddings. Despite this purely geometric origin, we show that its curvature is semantically meaningful: On chess reasoning tasks, the curvature couples to the world model of an off-the-shelf instruction-tuned model, with transformations clustering by board region and respecting piece importance. Our findings suggest that token space geometry directly reflects how models internally represent problems.
| Comments: | 12 Pages, 10 Figures, 2 Appendices. To appear in Proceedings of ICML 2026 |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.04899 [cs.LG] |
| (or arXiv:2605.04899v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.04899 arXiv-issued DOI via DataCite (pending registration) |
From: Albert Felix Modenbach [view email]
[v1]
Wed, 6 May 2026 13:28:16 UTC (1,489 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。