Abstract:Federated fine-tuning with low-rank adaptation (LoRA) communicates only two low-rank matrices instead of the full model, but existing methods typically fix the LoRA rank in advance as a manually tuned hyperparameter. In wireless networks, however, the rank determines both adaptation capacity and uplink payload, while the deliverable payload varies with the fading channel. To address this coupling, we formulate wireless federated LoRA fine-tuning as a two-timescale design that separates the \emph{offline-optimized rank}, i.e., the rank of the shared LoRA structure selected before training from statistical channel information, from the per-iteration sparsification and bandwidth decisions adapted to instantaneous CSI. For per-iteration adaptation, we propose sparsified orthogonal fine-tuning (\textbf{SOFT}), which promotes near-orthogonality among rank components so that the product of the corresponding column and row norms approximates each component's singular value. This SVD-free score guides component-wise payload allocation and within-component entry selection without forming the full matrix product. We further derive a convergence bound linking the two timescales and develop a two-stage federated algorithm (\textbf{TSFA}) that selects the offline-optimized rank offline and jointly optimizes sparsification and bandwidth online via Lyapunov optimization.
From: Bumjun Kim [view email]
[v1]
Thu, 1 May 2025 06:15:38 UTC (471 KB)
[v2]
Tue, 24 Mar 2026 08:50:28 UTC (356 KB)
[v3]
Sun, 23 Aug 2026 03:29:51 UTC (331 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。