“By the end of 2024, we’re aiming to continue to grow our infrastructure build-out that will include 350,000 NVIDIA H100 GPUs as part of a portfolio that will feature compute power equivalent to nearly 600,000 H100s.”
Meta built two clusters of 24,576 H100s each, one on Ethernet and one on InfiniBand, to train Llama 3. Out of the box the big clusters ran at 10% to 90% utilization, while small ones hit 90% without trying. Fixing it took scheduler, routing and NCCL changes. The bottleneck at this scale is the network, and NVIDIA sells that too.