Abstract:We propose DualOptim+, a novel optimization framework for improving machine unlearning in large language models. It introduces a base state to capture common representations shared by forgetting and retaining objectives and delta states to preserve objective-specific residuals. This architecture allows the optimizer to adaptively bridge shared and decoupled states based on the directional conflict between forgetting and retaining gradients. We further introduce DualOptim+ 8bit, a quantized variant that reduces memory overhead without compromising performance. Extensive experiments across fictitious and real-world unlearning, safety alignment, and multi-task learning tasks demonstrate that DualOptim+ consistently achieves a superior trade-off between different objectives. Codes are available at this https URL.
| Comments: | Accepted by ICML 2026 |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.21539 [cs.LG] |
| (or arXiv:2605.21539v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.21539 arXiv-issued DOI via DataCite |
From: Xuyang Zhong [view email]
[v1]
Wed, 20 May 2026 07:45:08 UTC (365 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。