“It took 65 days to train the model on a cluster of 800 A100 graphics cards and 1.7 TB of online texts, books, and countless other sources in both English and Russian.”
Yandex trained a GPT-class model on 800 A100s and gave away the weights under Apache 2.0. OpenAI keeps GPT-3 behind a paid API. A Russian search company is now more open than the lab with “Open” in its name. The weights are free. The hardware to run a 100 billion parameter model is not.