Abstract:The long-output context generation of large reasoning models enables extended chain of thought (CoT) but also drives rapid growth of the key-value (KV) cache, quickly overwhelming GPU memory. To address this challenge, we propose ThinKV, a thought-adaptive KV cache compression framework. ThinKV is based on the observation that attention sparsity reveals distinct thought types with varying importance within the CoT. It applies a hybrid quantization-eviction strategy, assigning token precision by thought importance and progressively evicting tokens from less critical thoughts as reasoning trajectories evolve. Furthermore, to implement ThinKV, we design a kernel that extends PagedAttention to enable efficient reuse of evicted tokens' memory slots, eliminating compaction overheads. Extensive experiments on DeepSeek-R1-Distill, GPT-OSS, and NVIDIA AceReason across mathematics and coding benchmarks show that ThinKV achieves near-lossless accuracy with less than 5% of the original KV cache, while improving performance with up to 5.8x higher inference throughput over state-of-the-art baselines.
| Comments: | ICLR 2026 (Oral) |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2510.01290 [cs.LG] |
| (or arXiv:2510.01290v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2510.01290 arXiv-issued DOI via DataCite |
From: Akshat Ramachandran [view email]
[v1]
Wed, 1 Oct 2025 04:09:02 UTC (8,514 KB)
[v2]
Thu, 7 May 2026 18:13:59 UTC (8,978 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。