Abstract:Counterfactual tuning (CFT) has emerged as a promising paradigm for Large Language Model (LLM) unlearning by training models to generate alternative fictitious knowledge in place of undesired content. However, in this work, we find that this paradigm still underperforms other paradigms in some aspects, and identify two previously overlooked pitfalls underlying this gap: (1) knowledge conflict, where mutual inconsistencies within counterfactual corpora induce conflicting gradients that disrupt parameter optimization, and (2) hallucination spillover, where fitting false targets instills a persistent fabrication bias, inflating hallucination rates on unrelated domains. To systematically diagnose these issues, we introduce RWKU+, an extended benchmark equipped with novel trade-off metrics and gradient-level diagnostic tools. Our work further discusses the limitations and overhead of the paradigm, aiming to provide insights and actionable guidance for more rigorous LLM unlearning research.
| Subjects: | Computation and Language (cs.CL); Cryptography and Security (cs.CR) |
| Cite as: | arXiv:2605.27083 [cs.CL] |
| (or arXiv:2605.27083v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.27083 arXiv-issued DOI via DataCite (pending registration) |
From: Xiaotian Ye [view email]
[v1]
Tue, 26 May 2026 14:34:14 UTC (188 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。