Abstract:Long-context inference is increasingly constrained by the KV cache: resident memory grows with context length, and decoding becomes limited by repeated High Bandwidth Memory (HBM) streaming rather than arithmetic. Existing methods such as eviction, windowing, quantization, and offloading reduce footprint, but often leave the critical-path bottleneck only partially addressed, especially when compressed states must still be reconstructed into dense vectors during decoding.
We present Spherical KV, a long-context inference method that treats KV allocation as a rate-distortion problem grounded in attention geometry for efficient decoding. The method is built on two ideas: (i) represent directional information cheaply in the decode hot loop, and (ii) allocate retention and precision according to estimated future utility. Its first component, Angle-Domain Attention (ADA), stores keys in a spherical parameterization consisting of a scalar radius and compact angle codes, and computes attention logits directly from these codes without reconstructing dense keys. This preserves a paged, block-local, fusion-friendly decode path and directly targets HBM traffic in realistic serving settings. Its second component, Rate-Distortion Retention (RDR), jointly chooses keep/drop decisions and precision tiers per token and head under a fixed budget, producing tier-homogeneous pages with lightweight metadata and coalesced reads. Together, ADA and RDR provide a deployment-oriented mechanism for reducing KV residency while preserving decode efficiency.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL); Information Theory (cs.IT) |
| ACM classes: | I.2.6; I.2.7 |
| Cite as: | arXiv:2605.18856 [cs.LG] |
| (or arXiv:2605.18856v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.18856 arXiv-issued DOI via DataCite |
From: Anay Chauhan [view email]
[v1]
Wed, 13 May 2026 18:48:48 UTC (10,532 KB)
[v2]
Tue, 26 May 2026 09:32:58 UTC (11,303 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。