“This approach achieves new top scores of 1.00 on the "Buffer Overflow" tests (from 0.05) and 0.76 on the "Advanced Memory Corruption" tests (from 0.24).”
Meta’s CyberSecEval 2 concluded LLMs were nowhere near useful for exploitation. It fed models a whole C++ file and demanded a crashing input with no reasoning allowed. Project Zero gave the same models a debugger, a Python sandbox and room to think, and scores jumped up to 20x. The benchmark measured the cage, not the animal.