“In the grand scheme of things, 5.42e24 is “not that big”.”
Google DeepMind’s Scaling Exponents Across Parameterizations and Optimizers reports over 10,000 LLM training runs to tune hyperparameters. Replicating it at $3 per H100-hour comes to about US$12.9 million. The epsilon heatmaps alone are worth US$3.2 million. Nobody outside a hyperscaler can check a result that costs eight figures to reproduce.