Abstract:Stochastic bilevel optimization (SBO) has become a standard framework for hyperparameter learning, data reweighting, representation learning, and data-mixture optimization in deep learning. Existing exact single-loop SBO methods and memory-efficient surrogate SBO methods either create severe memory pressure for large lower-level neural networks or lack competitive convergence guarantees under standard assumptions. In this paper, we propose BROS, a memory-efficient single-loop SBO method with the same convergence rate order as exact single-loop SBO methods. BROS performs lower and auxiliary updates in randomized subspaces with a Rademacher bi-probe correction that recovers an unbiased Hessian-action estimator. We prove that BROS preserves the $\mathcal O(\varepsilon^{-2})$ sample complexity of MA-SOBA for finding an $\varepsilon$-stationary point under only standard assumptions. Experiments on hyper-data cleaning, data-mixture learning, hyper-representation learning, and ViT sample reweighting show that BROS reduces peak memory by up to 44.9% while closely matching full-space baseline performance.
| Subjects: | Machine Learning (cs.LG); Optimization and Control (math.OC) |
| MSC classes: | 90C30, 90C15, 68T07 |
| Cite as: | arXiv:2605.10288 [cs.LG] |
| (or arXiv:2605.10288v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.10288 arXiv-issued DOI via DataCite (pending registration) |
From: Hengrui Zhang [view email]
[v1]
Mon, 11 May 2026 09:50:10 UTC (1,262 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。