Abstract:This article initiates the study of a basic question about model pruning. Given a vector $s$ of importance scores assigned to model components, how many of the scored components could be discarded without sacrificing performance? We propose Effective Model Pruning (EMP), which derives the desired sparsity directly from the score distribution using the notion of effective sample size from particle filtering, also known as the inverse Simpson index.
Rather than prescribe a pruning criterion, EMP supplies a universal adaptive threshold derived from the distribution of the score $s$ over the model components: EMP maps $s$ to a number $N_{eff}=N_{eff}(s)$, called the effective sample size. The $N-N_{eff}$ lowest scoring components are discarded. A tight lower bound on the effective mass $s_{eff}$ (the sum of retained normalized scores) in terms of $N_{eff}$ is derived. This process yields models with a provable upper bound on the loss change relative to the original dense model. Numerical experiments are performed demonstrating this phenomenon across a variety of network architectures including MLPs, CNNs, Transformers, LLMs, and KAN. It is also shown that EMP addresses a rich set of pruning criteria such as weight magnitude, attention score, KAN importance score, and even feature-level signals such as image pixels.
| Comments: | 18 pages, 4 figures. Accepted at ICML 2026 (Spotlight) |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2509.25606 [cs.LG] |
| (or arXiv:2509.25606v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2509.25606 arXiv-issued DOI via DataCite |
From: Yixuan Wang [view email]
[v1]
Tue, 30 Sep 2025 00:01:22 UTC (1,768 KB)
[v2]
Wed, 28 Jan 2026 01:55:46 UTC (1,879 KB)
[v3]
Wed, 20 May 2026 11:03:34 UTC (1,830 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。