Abstract:Scaling factors in residual branches have emerged as a prevalent method for boosting neural network performance, especially in normalization-free architectures. While prior work has primarily examined scaling effects from an optimization perspective, this paper investigates their role in residual architectures through the lens of generalization theory. Specifically, we establish that wide residual networks (ResNets) with constant scaling factors become asymptotically unlearnable as depth increases. In contrast, when the scaling factor exhibits rapid depth-wise decay combined with early stopping, over-parameterized ResNets achieve minimax-optimal generalization rates. To establish this, we demonstrate that the generalization capability of wide ResNets can be approximated by kernel regression associated with the Neural Tangent Kernel (NTK). Our theoretical findings are validated through experiments on synthetic data and real-world classification tasks, including MNIST and CIFAR-100.
| Comments: | Accepted by ICML. This version incorporates content from the preprint arXiv:2305.18506. The contributors of the relevant content have consented to its inclusion and have been listed as authors |
| Subjects: | Machine Learning (cs.LG); Statistics Theory (math.ST) |
| Cite as: | arXiv:2403.04545 [cs.LG] |
| (or arXiv:2403.04545v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2403.04545 arXiv-issued DOI via DataCite |
From: Zixiong Yu [view email]
[v1]
Thu, 7 Mar 2024 14:40:53 UTC (174 KB)
[v2]
Thu, 26 Mar 2026 09:39:33 UTC (441 KB)
[v3]
Mon, 25 May 2026 16:06:08 UTC (420 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。