Abstract:Model merging aims to combine multiple models into one without additional training. Naïve parameter-space averaging can be fragile under architectural symmetries, as their geometry does not take them into account. In this work we show that not only the geometry, but also the averaging procedure itself, must be symmetry-invariant to achieve symmetry-aware merges. Consequently, we propose a general solution: merging as Fréchet averaging, i.e., selecting parameters that minimize a sum of geodesic distances on an appropriate manifold. In this view, the key design choice is the overall geometry, i.e., the choice of metric, manifold, and distance approximation, that determines what it means for two models to be "close". We show that Fréchet averaging, combined with simplifying assumptions, contains Fisher merging. Building on this, we examine the particular case of low-rank adapters (LoRA), whose symmetries induce a distinct geometry: that of a quotient manifold. We outline the limitations of current LoRA merging methods, propose a practical algorithm for this setting, and show how they compare with other commonly used approaches.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2604.27155 [cs.LG] |
| (or arXiv:2604.27155v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2604.27155 arXiv-issued DOI via DataCite (pending registration) |
From: Marvin F. Da Silva [view email]
[v1]
Wed, 29 Apr 2026 20:02:08 UTC (478 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。