False positive identification rate
Posted Apr 17, 2026 7:03 UTC (Fri) by daenzer (subscriber, #7050)In reply to: False positive identification rate by iabervon
Parent article: The role of LLMs in patch review
Is there any information about how leakage was prevented when measuring that ~50% rate, i.e. how it was ensured the LLM didn't have knowledge of the corresponding fixes?
Per https://www.normaltech.ai/p/scientists-should-use-ai-as-a... leakage causes the effectiveness of machine-learning models to be overestimated in many scientific papers.