“This congestion immediately impacted the availability of real-time monitoring data for our internal operations teams, which impaired their ability to find the source of congestion and resolve it.”
An automated scaling job flooded AWS’s internal network. That took out the monitoring AWS uses to find problems. It also broke the status page that tells customers there is a problem. The network didn’t fully recover until 2:22 PM, almost seven hours in. AWS sells resilience and couldn’t see its own fire.