























Abstract:Regression Mean (RegMean), an approach that formulates model merging as a linear regression problem, aims to find the optimal weights for each linear layer in the merged model by minimizing the discrepancy in predictions between the merged and candidate models. RegMean provides a precise closed-form solution for the merging problem; therefore, it offers explainability and computational efficiency. However, RegMean merges each linear layer independently, overlooking how the features and information in earlier layers propagate through deeper layers and influence the final predictions of the merged model. Here, we introduce RegMean++, a simple yet effective alternative to RegMean, that explicitly incorporates both intra-layer and cross-layer dependencies between merged models' layers into RegMean's objective. By accounting for these dependencies, RegMean++ better captures the behaviors of the merged model. Extensive experiments demonstrate that RegMean++ consistently outperforms RegMean across diverse settings, including in-domain (ID) and out-of-domain (OOD) generalization, sequential merging, large-scale tasks, and robustness under several types of distribution shifts. Furthermore, RegMean++ achieves competitive performance across diverse settings compared to various advanced model merging methods.
| Comments: | Accepted by Transactions on Machine Learning Research |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2508.03121 [cs.LG] |
| (or arXiv:2508.03121v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2508.03121 arXiv-issued DOI via DataCite |
From: Tien Dang [view email]
[v1]
Tue, 5 Aug 2025 06:08:26 UTC (781 KB)
[v2]
Thu, 11 Dec 2025 04:49:06 UTC (924 KB)
[v3]
Sat, 25 Apr 2026 21:24:49 UTC (1,050 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。