“We previously had a test log fail when we ran into a 16TiB limit in MySQL.”
Let’s Encrypt issues over four million certificates a day, and each one goes into two Certificate Transparency logs. Its Oak log holds over 700 million entries in MySQL shards of 5 to 10 terabytes, and readers crawling the log keep knocking it over. Sunlight serves the tree as static tiles from object storage and drops the leader election cluster for a single writer. The safety net of the Web PKI was running on the most expensive storage in the cloud, and nobody fixed it until the bill forced the issue.