Abstract:Mixed-precision computations are a hallmark of the current stage of AI, driving the progress in large language models towards efficient, locally deployable solutions. This article addresses the floating-point computation of compositionally-rich functions, concentrating on transformer inference. Based on the rounding error analysis of a composition $f(g(\mathrm{x}))$, we provide an adaptive strategy that selects a small subset of components of $g(\mathrm{x})$ to be computed more accurately while all other computations can be carried out with lower accuracy. We then explain how this strategy can be applied to different compositions within a transformer and illustrate its overall effect on transformer inference. We study the effectiveness of this algorithm numerically on GPT-2 models and demonstrate that already very low recomputation rates allow for improvements of up to two orders of magnitude in accuracy.
| Comments: | Major revision |
| Subjects: | Machine Learning (cs.LG); Numerical Analysis (math.NA) |
| Cite as: | arXiv:2601.21623 [cs.LG] |
| (or arXiv:2601.21623v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2601.21623 arXiv-issued DOI via DataCite |
From: Stanislav Budzinskiy [view email]
[v1]
Thu, 29 Jan 2026 12:26:00 UTC (358 KB)
[v2]
Thu, 7 May 2026 16:10:14 UTC (86 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。