“The training for Phi-2 took 14 days on 96 A100 GPUs.”
Phi-2 has 2.7 billion parameters and Microsoft says it beats Llama-2-70B, a model 25x larger, on coding and math. The trick is “textbook-quality” data, much of it synthetic and written to teach. It also matches Google’s recently announced Gemini Nano 2, a point Microsoft made sure to include. Microsoft’s own result says data beats size, an awkward finding for a company renting out GPUs by the hour.