Abstract:We present AutoCompress, a transformer compression method motivated by an empirical finding: in small transformers, Layer 0 carries disproportionately high task-critical information, with an NTK-based importance score of 3.6 compared to a maximum of 0.054 for all other layers -- a gap of over 60x. Based on this finding, we propose Critical Layer Isolation (CLI), an architecture that protects Layer 0 at full dimensionality, compresses all intermediate layers through a learned bottleneck, and restores the full dimension at the final layer. Applied to GPT-2 Medium (354.8M parameters), CLI-GPT2 achieves 204.5 perplexity on WikiText-103 with only 143.8M parameters -- a 2.47x compression ratio and 59.5% parameter reduction. Crucially, an ablation study demonstrates that a uniform bottleneck baseline of comparable size achieves only 571.8 perplexity under identical training conditions, confirming that the architectural decision to protect Layer 0 -- rather than simply reducing model size -- is the primary driver of performance. Code and checkpoints are publicly available.
| Comments: | 6 pages, 2 tables. Code available at this https URL |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2604.22786 [cs.LG] |
| (or arXiv:2604.22786v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2604.22786 arXiv-issued DOI via DataCite |
From: Archit Thorat [view email]
[v1]
Sat, 4 Apr 2026 22:14:24 UTC (269 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。