“NB, while neo can technically run a training step at 200B+ parameters, it is very inefficient at those scales.”
GPT-3 lives behind an API you rent by the token. EleutherAI trained its own models on The Pile and posted 1.3B and 2.7B weights anyone can download. They are a fraction of GPT-3’s size. They are also yours to run, inspect and break.