Abstract:Model merging combines knowledge from separately fine-tuned models, yet the factors driving its success remain poorly understood. While recent work treats mergeability as an intrinsic property of the models, we show with an architecture-agnostic framework that it fundamentally depends on both the merging method and the partner tasks. Using L1-regularized linear optimization over a set of interpretable pairwise metrics (e.g., gradient $L_2$ distance), we uncover properties correlating with post-merge normalized accuracy across five merging methods. We find architecture- and method-specific variation in success drivers (64.0% average top-5 metric overlap; 79.3% sign agreement), with certain methods, notably TIES, exhibiting distinct ``fingerprints'' that diverge from the broader consensus. Crucially, however, \textit{gradient alignment} metrics consistently emerge as the most fundamental signals of compatibility. These findings provide a diagnostic foundation for understanding mergeability and motivate future merge-aware fine-tuning strategies.
| Comments: | 9 pages of main paper, 3 figures in the main paper, 4 tables in the main paper, many more figures and tables in the appendix |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2601.22285 [cs.LG] |
| (or arXiv:2601.22285v5 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2601.22285 arXiv-issued DOI via DataCite |
From: Luca Zhou [view email]
[v1]
Thu, 29 Jan 2026 20:00:26 UTC (271 KB)
[v2]
Mon, 2 Feb 2026 07:07:31 UTC (262 KB)
[v3]
Fri, 6 Feb 2026 17:53:23 UTC (271 KB)
[v4]
Fri, 10 Apr 2026 09:47:42 UTC (272 KB)
[v5]
Fri, 1 May 2026 17:12:01 UTC (210 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。