
























Abstract:Modern LLM APIs often reveal only top-$K$ logit scores and censor the remaining vocabulary. We study the per-position distribution-recovery limits of this access model. For censoring threshold $\tau$, the compatible teacher distributions form an identified set whose total-variation diameter is exactly $U_K=(V-K)\exp(\tau)/(Z_A+(V-K)\exp(\tau))$, where $Z_A$ is the observed partition function. For KL recovery, we give a computable binary-endpoint lower bound and an asymptotically matching small-ambiguity upper bound, with an extension to reference-aware attackers. Experiments on a Qwen3 math-reasoning teacher reveal a layered extraction hierarchy: on-task top-$K$ distillation recovers 12% of private capability, full-logit distillation recovers 56% despite 99% KL closure, and generation-based extraction recovers 96%. Top-$K$ censoring therefore limits per-position distribution recovery but does not by itself prevent capability extraction, separating fidelity from transfer in prompt-only logit distillation.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.10407 [cs.LG] |
| (or arXiv:2605.10407v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.10407 arXiv-issued DOI via DataCite (pending registration) |
From: Wenhua Nie [view email]
[v1]
Mon, 11 May 2026 11:43:48 UTC (52 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。