Abstract:We study the max-margin solutions reached by mirror flow in deep neural networks with homogeneous activation functions. Extending classical results on gradient flow, we derive a novel balance equation for mirror flow from convex duality, enabling a characterization of the horizon function governing the induced margin. We further establish max-margin characterizations together with convergence rates and norm growth estimates. Finally, we support our theory through experiments on synthetic datasets and standard vision tasks. Concretely, we show that: (1) distinct non-homogeneous mirror maps can induce the same max-margin solution; (2) convergence can be extremely slow, including exponentially slow regimes; and (3) although all considered mirror maps exhibit feature learning, they can produce markedly different representations, ranging from sparse to dense neuron activations. Together, these results provide a unified perspective on sparse and dense feature learning in homogeneous neural networks, highlighting how mirror maps shape both optimization dynamics and the geometry of the learned classifiers.
| Comments: | 36 pages, 14 figures |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.19458 [cs.LG] |
| (or arXiv:2605.19458v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.19458 arXiv-issued DOI via DataCite (pending registration) |
From: Tom Jacobs [view email]
[v1]
Tue, 19 May 2026 07:10:18 UTC (599 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。