“o3 finds the kerberos authentication vulnerability in the benchmark in 8 of the 100 runs. In another 66 of the runs o3 concludes there is no bug present in the code (false negatives), and the remaining 28 reports are false positives.”

Sean Heelan found a remote Linux kernel zeroday with plain o3 API calls and no tooling. On a known bug it scored 8 hits and 28 false positives in 100 runs. With the larger codebase he puts the signal to noise at about 1 to 50. That works for an expert who can sort the junk. Maintainers getting raw output from everyone else just get the junk.