Abstract:Differentially private (DP) transfer learning, i.e., fine-tuning a pretrained model on private data, is the current state-of-the-art approach for training large models under privacy constraints. We focus on two key hyperparameters in this setting: the clipping bound $C$ and batch size $B$. We show a clear mismatch between the current theoretical understanding of how to choose an optimal $C$ (stronger privacy requires smaller $C$) and empirical outcomes (larger $C$ performs better under strong privacy), caused by changes in the gradient distributions. Assuming a limited compute budget (fixed epochs), we demonstrate that the existing heuristics for tuning $B$ do not work, while cumulative DP noise better explains whether smaller or larger batches perform better. We also highlight how the common practice of using a single $(C,B)$ setting across tasks can lead to suboptimal performance. We find that performance drops especially when moving between loose and tight privacy and between plentiful and limited compute, which we explain by analyzing clipping as a form of gradient re-weighting and examining cumulative DP noise.
| Comments: | ICLR 2026 |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2510.20616 [cs.LG] |
| (or arXiv:2510.20616v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2510.20616 arXiv-issued DOI via DataCite |
From: Aki Rehn [view email]
[v1]
Thu, 23 Oct 2025 14:48:03 UTC (2,801 KB)
[v2]
Fri, 17 Apr 2026 12:23:04 UTC (6,989 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。