“Meta did this because a single failed GPU would take out an entire rack and interrupt the training run.”
Meta keeps more than 11 percent of the GPUs in Prometheus dead or idle. Every 72-GPU rack runs 64 and holds 8 in reserve to replace the ones that die. That cut training interruptions by 99 percent, which tells you how often these chips fail. Nvidia gets paid for the spares too.