“the overnight shift consisted of security and an unaccompanied technician who had only been on the job for a week.”
Flexential ran on generators without telling Cloudflare, then a ground fault knocked out the utility feed and all 10 generators. Batteries rated for 10 minutes died in 4. The access control system had no battery backup, so staff couldn’t easily get to the gear. Cloudflare’s own failure is worse. It never tested losing the whole PDX-04 building, and Kafka and ClickHouse lived only there. Four years into a high availability project, the dashboard was down for 40 hours.