“Our data generation process results in 52K unique instructions and the corresponding outputs, which costed less than $500 using the OpenAI API.”
Stanford fine-tuned Meta’s LLaMA 7B on 52,000 examples written by OpenAI’s text-davinci-003. Data cost under $500, training took 3 hours on eight A100s for under $100. The result behaves a lot like the model it copied. OpenAI’s terms ban using its output to build competitors, which is why Alpaca is research-only. Every closed model now doubles as a $600 teacher for its own rivals.