“They are doing things with $100 and 13B params that we struggle with at $10M and 540B.”
A Google researcher wrote it, and it leaked through a public Discord server. Meta released LLaMA without the weights, and within weeks hobbyists had 13B models that users preferred or rated equal to ChatGPT more than half the time, for about $100 in training. Google is spending $10 million to stay slightly ahead. The moat Google and OpenAI are selling to investors cost a weekend and a gaming GPU to cross.