“At the beginning of 2022, it had 177 nodes with trillions of messages.”
Discord went from 12 Cassandra nodes in 2017 to 177 by 2022 and spent its days firefighting garbage collection pauses and hot partitions. ScyllaDB cut that to 72 nodes, and historical fetch p99 dropped from 40-125ms to 15ms. The migration ran at up to 3.2 million messages per second. Java’s garbage collector was the most expensive employee on the team.