Abstract:Zeroth-Order (ZO) optimization has emerged as a promising solution for fine-tuning LLMs under strict memory constraints, as it avoids the prohibitive memory cost of storing activations for backpropagation. However, existing ZO methods typically employ isotropic perturbations, neglecting the rich structural information available during the forward pass. In this paper, we identify a crucial link between gradient formation and activation structure: the gradient of a linear layer is confined to the subspace spanned by its input activations. Leveraging this insight, we propose Activation-Guided Zeroth-Order optimization (AGZO). Unlike prior methods, AGZO extracts a compact, activation-informed subspace on the fly during the forward pass and restricts perturbations to this low-rank subspace. We provide a theoretical framework showing that AGZO optimizes a subspace-smoothed objective and provably yields update directions with higher cosine similarity to the true gradient than isotropic baselines. Empirically, we evaluate AGZO on Qwen3 and Pangu models across various benchmarks. AGZO consistently outperforms state-of-the-art ZO baselines and significantly narrows the performance gap with first-order fine-tuning, while maintaining almost the same peak memory footprint as other ZO methods.
| Comments: | 21 pages in total, including 9 pages of main text, with 4 figures and 3 tables Accepted by ICML 2026 |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2601.17261 [cs.LG] |
| (or arXiv:2601.17261v4 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2601.17261 arXiv-issued DOI via DataCite |
From: Wei Lin [view email]
[v1]
Sat, 24 Jan 2026 02:28:15 UTC (154 KB)
[v2]
Wed, 28 Jan 2026 04:26:27 UTC (156 KB)
[v3]
Tue, 10 Feb 2026 00:58:44 UTC (159 KB)
[v4]
Fri, 22 May 2026 07:39:05 UTC (161 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。