Abstract:We propose a scalable tensorization framework for neural network compression based on slice-wise feature distillation. Unlike conventional tensor decomposition methods that rely on costly global finetuning, our approach decomposes the network into slices consisting of either individual layers or blocks (e.g., convolutional layers or MLPs), or small groups of consecutive layers, and tensorizes each slice independently to reproduce the intermediate representations of the original pretrained model. This modular strategy improves accuracy recovery, reduces data requirements, and enables efficient parallel optimization. Experiments on ResNet-34 show significant gains over conventional global tensorization, achieving near-lossless compression at moderate compression rates with faster optimization. Results on GPT-2 XL further demonstrate the scalability of the method and its applicability to large-scale models, particularly in distributed settings.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.19842 [cs.LG] |
| (or arXiv:2605.19842v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.19842 arXiv-issued DOI via DataCite (pending registration) |
From: Safa Hamreras [view email]
[v1]
Tue, 19 May 2026 13:37:53 UTC (3,044 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。