Abstract:Large language models are trained on massive corpora of web data, which may include private data, copyrighted material, factually inaccurate data, or data that degrades model performance. Eliminating the influence of such problematic datapoints on a model through complete retraining -- by repeatedly pretraining the model on datasets that exclude these specific instances -- is computationally prohibitive. To address this, unlearning algorithms have been proposed, that aim to eliminate the influence of particular datapoints at a low computational cost, while leaving the rest of the model intact. However, precisely unlearning the influence of data on a large language model has proven to be a major challenge. In this work, we propose a new algorithm, MSA (Model State Arithmetic), for unlearning datapoints in large language models. MSA utilizes prior model checkpoints -- artifacts that record model states at different stages of pretraining -- to estimate and counteract the effect of targeted datapoints. Our experimental results show that MSA achieves competitive performance and often outperforms existing machine unlearning algorithms across multiple benchmarks, models, and evaluation metrics, suggesting that MSA could be an effective approach towards more flexible large language models that are capable of data erasure.
| Comments: | Accepted to ICLR 2026 |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2506.20941 [cs.LG] |
| (or arXiv:2506.20941v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2506.20941 arXiv-issued DOI via DataCite |
From: Keivan Rezaei [view email]
[v1]
Thu, 26 Jun 2025 02:16:16 UTC (1,964 KB)
[v2]
Mon, 26 Jan 2026 15:38:28 UTC (502 KB)
[v3]
Mon, 27 Apr 2026 18:06:07 UTC (499 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。