↓ Skip to main content

“In addition, our particular load conditions triggered a pathological performance issue in BoltDB.”

Daniel Sturmanblog.roblox.com ↗

Roblox was down for 73 hours because one Consul cluster sat under everything and choked. Turning on a newish streaming feature under heavy load did the rest. The same service discovery system the whole platform relied on also ran the monitoring they needed to debug it. Fifty million daily players found out what a single point of failure looks like.