“Our new benchmark approach to MMLU enables Gemini to use its reasoning capabilities to think more carefully before answering difficult questions, leading to significant improvements over just using its first impression.”
Google’s headline number is a 90.0% MMLU score for Gemini Ultra, “the first model to outperform human experts.” It got there with a new benchmark approach Google picked for itself. Ultra isn’t shipping, so Bard gets the smaller Gemini Pro today, and Ultra waits on safety checks until “early next year.” Google launched a scoreboard and kept the winning model in the back room.