“This was a pretty scary hole: you could ask it to run anything, and those commands would be run as root.”
A performance tool ran on every production machine as root and accepted arbitrary pre and post commands over the network. Nobody needed special permissions to use it. It logged neither the request nor the command. The hole was found in leftover meeting time with one database query, and there was no way to tell who else had found it first.