Plateauing capabilities won't help

But I mean significantly reduce the number of new vulns that are discoverable with $20 of tokens and a prompt that says "find vuln plz".

Which sounds fairly reasonable. I don't think attackers are going to have access to notably better quality LLM support than defenders are, and the defenders have the advantage that they never have to merge anything that doesn't pass the LLM vulnerability test. If anything, I would expect something like what's happening now, where Anthropic is holding off releasing their next generation LLM until they've had a chance to patch vulnerabilities it finds. The one big thing FOSS has going in its favor is that the LLM companies use it extensively themselves and will want to make sure the stuff they're using isn't full of easily exploitable vulnerabilities.