























Abstract:Zeroth-order optimization has emerged as a promising approach for fine-tuning large language models under differential privacy (DP) and memory constraints. While privacy amplification by iteration (PABI) provides convergent DP bounds for first-order methods, establishing similar guarantees for zeroth-order methods remains an open problem. First-order PABI analysis relies on the fact that gradients are perturbed with isotropic noise, allowing privacy bounds to be iteratively tracked via shifted Rényi divergence. In contrast, DP zeroth-order methods inject scalar noise along random update directions to maintain utility. This anisotropic update fails standard shifted divergence frameworks, as the global Lipschitz property no longer holds almost surely. We provide the first convergent hidden-state DP bound for zeroth-order optimization by proposing a hybrid noise mechanism and a novel coupling analysis. We bypass the purely shifted-divergence approach by constructing a coupled auxiliary process, which circumvents the global Lipschitz barrier and yields a convergent privacy bound. Furthermore, our results induce better DP zeroth-order algorithmic designs that are previously unknown to the literature.
| Comments: | Preprint. To appear at ICML 2026 |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2506.00158 [cs.LG] |
| (or arXiv:2506.00158v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2506.00158 arXiv-issued DOI via DataCite |
From: Eli Chien [view email]
[v1]
Fri, 30 May 2025 18:55:32 UTC (262 KB)
[v2]
Fri, 1 May 2026 05:17:48 UTC (509 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。