“The issue with this change was that it did not have appropriate error handling nor was it feature flag protected. Without the appropriate error handling, the null pointer caused the binary to crash.”
A policy with blank fields replicated worldwide in seconds and crashed Google Cloud’s API authorization in every region. The bad code had shipped two weeks earlier with no feature flag and no error handling. Restarts then hammered Spanner because there was no backoff. Google’s status page was down too, so customers learned about it an hour late.