Abstract:Weak-to-strong generalization is a phenomenon in post-training whereby a strong student model, when finetuned solely with feedback from a weaker teacher, can not only surpass the teacher, but can improve upon its own capabilities. Recent work of Burns et al. (2023) demonstrated that this can occur in the setting of frontier language models, and subsequently there has been a flurry of both empirical work trying to exploit this phenomenon, as well as theoretical work attempting to understand it. In this work, we demonstrate that weak-to-strong generalization occurs in standard linear logistic regression, under mild distributional assumptions on the data. In fact, we show that this happens for most student-teacher pairs, suggesting that weak-to-strong generalization is in fact \emph{almost inevitable}, even in this basic setting. Notably, our setting does not require the student to be more expressive or have more model capacity in any way compared to the teacher, which runs contrary to the prevailing theoretical belief that a mismatch in model capacity is a central mechanism to weak-to-strong generalization.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.05742 [cs.LG] |
| (or arXiv:2605.05742v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.05742 arXiv-issued DOI via DataCite (pending registration) |
From: Dutch Hansen [view email]
[v1]
Thu, 7 May 2026 06:33:25 UTC (1,256 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。