


























Abstract:To drive progress in science and engineering, large language models (LLMs) must be able to process large amounts of numerical data and solve long calculations efficiently. This is currently only possible through the use of external tools or extensive reasoning chains, either weakening the numerical representations of LLMs or limiting the length of problems they can solve. We show that frontier LLMs require excessive amounts of reasoning tokens to solve even basic calculations, which is exacerbated by their tokenization strategies that split single numbers into multiple tokens. This motivates the need for efficient and effective single-token number encodings. We introduce a set of desiderata for such encodings and show that existing approaches fail to fulfill them. To address these shortcomings, we propose BitTokens, a novel encoding strategy that represents any number as a single token using its IEEE 754 binary floating-point representation. Through extensive experiments we show that our BitTokens allow even small language models to learn algorithms that solve basic arithmetic operations nearly perfectly. This newly gained efficiency could expand the length and complexity of problems language models can solve.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2510.06824 [cs.LG] |
| (or arXiv:2510.06824v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2510.06824 arXiv-issued DOI via DataCite |
From: Linus Kreitner [view email]
[v1]
Wed, 8 Oct 2025 09:48:11 UTC (606 KB)
[v2]
Wed, 20 May 2026 14:19:31 UTC (6,262 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。