Abstract:Fine-tuning adapts a pre-trained model to downstream tasks using a small amount of labeled data. Low-Rank Adaptation (LoRA) is an efficient fine-tuning method that reduces memory and computation costs while often achieving performance close to full fine-tuning. Despite its widespread use, the theoretical behavior of LoRA is not yet well understood. In this paper, we study LoRA in a simple linear regression setting and compare its excess risk with that of full fine-tuning. Our analysis identifies regimes in which LoRA achieves lower excess risk than full fine-tuning in both overdetermined and underdetermined settings. Specifically, our theory predicts that LoRA can outperform full fine-tuning when the difference between the pretraining and the downstream tasks is effectively low-rank. We further show how the choice of LoRA rank affects generalization performance, explaining why using a very small rank can improve test accuracy in certain settings, even though it limits model expressivity. Finally, we support our theoretical results with experiments on practical tasks, suggesting that the identified tradeoffs and insights extend beyond linear regression.
| Comments: | Preprint |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.19018 [cs.LG] |
| (or arXiv:2605.19018v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.19018 arXiv-issued DOI via DataCite (pending registration) |
From: Rotem Mulayoff [view email]
[v1]
Mon, 18 May 2026 18:40:24 UTC (3,107 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。