Abstract:Speculative decoding and dynamic sparse attention are two complementary approaches for accelerating long-context LLM inference: the former amortizes target-model execution across multiple verifier queries, while the latter reduces each query's KV-cache working set. Directly combining them, however, exposes a structural mismatch: speculative verification relies on cross-query commonality, whereas dynamic sparse attention assigns query-specific sparse layouts. This mismatch limits KV-block reuse, amplifies NSA's branch-wise overheads, and makes verification strategy selection input- and regime-dependent. We present SSV, a sparse speculative-verification framework that turns dynamic sparse attention into a verification-oriented workload. SSV combines overlap-aware grouped-query execution, refresh/reuse-based NSA kernel fusion, and profile-guided prompt-adaptive orchestration to improve cross-query reuse, reduce selected-index and branch-fusion overheads, and select effective draft-verification strategies under user-specified precision classes. Experiments on NVIDIA H100 GPUs show that SSV achieves up to 3.49x end-to-end throughput over autoregressive NSA decoding and up to 6.86x kernel speedups for sparse speculative verification.
| Subjects: | Operating Systems (cs.OS) |
| Cite as: | arXiv:2605.19893 [cs.OS] |
| (or arXiv:2605.19893v2 [cs.OS] for this version) | |
| https://doi.org/10.48550/arXiv.2605.19893 arXiv-issued DOI via DataCite |
From: Yuhang Zhou [view email]
[v1]
Tue, 19 May 2026 14:24:27 UTC (2,447 KB)
[v2]
Wed, 20 May 2026 15:53:57 UTC (2,446 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。