Abstract:Fine-tuning large language models (LLMs) on resource-constrained clients remains a challenging problem. Recent works have fused low-rank adaptation (LoRA) techniques with federated fine-tuning to mitigate challenges associated with client model sizes and data scarcity. Still, the heterogeneity of resources remains a critical bottleneck: while higher-rank modules generally enhance performance, varying client capabilities constrain LoRA's feasible rank range. Existing approaches attempting to resolve this issue either lack analytical justification or impose additional computational overhead, leaving a wide gap for efficient and theoretically-grounded solutions. To address these challenges, we propose federated sketching LoRA (FSLoRA), which leverages a sketching mechanism to enable clients to selectively update submatrices of global LoRA modules maintained by the server. By adjusting the sketching ratios, which determine the ranks of the submatrices on the clients, FSLoRA flexibly adapts to client-specific communication and computational constraints. We provide a rigorous convergence analysis of FSLoRA that characterizes how the sketching ratios affect the convergence rate. Through extensive experiments, we demonstrate that FSLoRA outperforms baselines and significantly improves training efficiency while preserving stable convergence.
| Comments: | We propose Federated Sketching LoRA (FSLoRA), a theoretically grounded methodology for collaborative LLM fine-tuning that retains LoRA's flexibility while adapting to the communication and computational capabilities of individual clients |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2501.19389 [cs.LG] |
| (or arXiv:2501.19389v4 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2501.19389 arXiv-issued DOI via DataCite |
From: Wenzhi Fang [view email]
[v1]
Fri, 31 Jan 2025 18:44:35 UTC (4,984 KB)
[v2]
Sun, 18 May 2025 02:24:58 UTC (5,205 KB)
[v3]
Sun, 28 Sep 2025 20:07:49 UTC (1,494 KB)
[v4]
Sat, 23 May 2026 21:24:55 UTC (1,503 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。