Abstract:Learned optimizers have been an active research topic over the past decade, with increasing progress toward practical, general-purpose optimizers that can serve as drop-in replacements for widely used methods like Adam. However, recent advances such as VeLO, which was meta-trained for 4000 TPU-months, remain largely inaccessible to the broader community, in part due to their reliance on JAX and the absence of user-friendly packages for independently using the optimizers after meta-training. To address this gap, we introduce PyLO, a PyTorch-based library that brings learned optimizers to the remaining ~70% of machine learning community via the familiar this http URL interface. Unlike prior work focused on limited-scale academic tasks, our emphasis is on applying learned optimization to real-world large-scale pre-training tasks. Our systems contribution includes CUDA-accelerated implementations of the small fc lopt(Metz et al., 2022a) and VeLO(Metz et al., 2022b) learned optimizers, achieving substantial performance gains, with training throughput on ViT-B/16 (batch size 32) increasing from 39.36 and 49.73 to 205.59 and 191.18 samples per second, respectively. PyLO has the versatility that allows us to easily combine learned optimizers with existing optimization tools such as learning rate schedules and weight decay. When doing so, we discover that learned optimizers can substantially benefit from it. Our code is available at this https URL
| Comments: | Accepted at MLSys Conference 2026 |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2506.10315 [cs.LG] |
| (or arXiv:2506.10315v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2506.10315 arXiv-issued DOI via DataCite |
From: Paul Janson [view email]
[v1]
Thu, 12 Jun 2025 03:00:46 UTC (9,704 KB)
[v2]
Mon, 10 Nov 2025 15:14:05 UTC (10,971 KB)
[v3]
Thu, 16 Apr 2026 22:47:24 UTC (10,849 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。