

























Authors:Yaoqi Chen, Jinkai Zhang, Baotong Lu, Qianxi Zhang, Chengruidong Zhang, Jing Liu, Jingjia Luo, Di Liu, Huiqiang Jiang, Qi Chen, Bailu Ding, Xiao Yan, Jiawei Jiang, Chen Chen, Mingxing Zhang, Cheng Li, Yuqing Yang, Fan Yang, Mao Yang
Abstract:Recent large language models (LLMs) are rapidly extending their context windows, yet inference throughput lags due to increasing GPU memory and bandwidth demands. This is because the key-value (KV) cache, an intermediate structure storing token representations, grows linearly with context length and requires an iterative linear scan for attention computation. A promising direction to accelerate long-context inference is to exploit attention's inherent sparsity by offloading the KV cache to CPU memory and retrieving only a small subset of tokens important to the current generation step. However, prior sparse attention approaches struggle to balance accuracy and retrieval cost due to varying sparsity patterns and inefficient GPU-CPU memory management.
We present RetroInfer, a vector storage engine that realizes a sparsity-based KV cache for long-context inference. RetroInfer introduces an Attention-aWare VEctor index (wave index), which fundamentally improves the tradeoff between attention accuracy and retrieval cost through tripartite attention approximation, accuracy-bound attention estimation, and segmented clustering. We also design the wave buffer, a GPU-CPU buffer manager that assigns computation and manages data across heterogeneous hardware. We evaluate RetroInfer across a range of models and workloads, demonstrating up to 4.4X decoding throughput over full attention at 120K context and up to 12.2X over sparse attention baselines at 1 million tokens -- all while preserving full-attention-level accuracy.
| Comments: | 16 pages; Accepted by VLDB 2026 |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2505.02922 [cs.LG] |
| (or arXiv:2505.02922v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2505.02922 arXiv-issued DOI via DataCite |
|
| Journal reference: | PVLDB, 19(5): 1016-1031, 2026 |
| Related DOI: | https://doi.org/10.14778/3796195.3796212
DOI(s) linking to related resources |
From: Baotong Lu [view email]
[v1]
Mon, 5 May 2025 18:01:17 UTC (676 KB)
[v2]
Mon, 30 Jun 2025 05:21:58 UTC (681 KB)
[v3]
Mon, 27 Apr 2026 10:13:35 UTC (600 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。