Abstract:Refactoring tools in popular Integrated Development Environments (IDEs) can introduce unintended behavioral changes or compilation errors, a persistent challenge that undermines developer trust in automated transformations. Traditional detection approaches rely on handcrafted preconditions, and static and dynamic analyses, yet remain limited in adaptability and can miss subtle correctness issues. This study examines the potential of foundation models to serve as oracles for detecting refactoring bugs in Java programs. We evaluate zero-shot prompting, without task-specific training, across 226 real refactoring bugs collected over more than a decade from widely used Java IDEs (IntelliJ-IDEA, Eclipse, and NetBeans), spanning 47 refactoring types. Our results indicate that foundation models can be effective for this task, although performance varies across models. In the first-run setting, GPT-OSS-20B achieved 80.5% accuracy, while GPT-5.4 reached 93.8%. We also evaluated other open-weight and proprietary models: Gemma-4-31B achieved the strongest result among open-weight models, and Gemini-3.1-Pro-Preview achieved the best overall result among all evaluated models. Metamorphic testing indicates that model predictions remain largely consistent under the tested semantics-preserving perturbations, but these results should be interpreted as robustness evidence rather than as evidence against memorization or data contamination. Beyond detection accuracy, foundation models can provide short explanations that may help support developer inspection, operate across refactoring types without explicitly encoded refactoring-specific rules, and may serve as lightweight triage aids in development workflows. Our findings suggest that foundation models can complement traditional refactoring checks by flagging suspicious transformations for developer inspection.
From: Rohit Gheyi [view email]
[v1]
Sun, 3 May 2026 23:31:18 UTC (1,644 KB)
[v2]
Wed, 1 Jul 2026 22:38:58 UTC (1,650 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。