“In addition, we’d like to note that engineers have worked to optimize prompts and few-shot samples for evaluations and reported higher scores for a newer GPT-4T model.”
Opus costs $15 per million input tokens and $75 per million output, and Anthropic says it beats its peers on MMLU, GPQA and GSM8K. The benchmark table compares against models currently on sale and skips the newer GPT-4T scores, which sit in a footnote. The 1 million token context exists, but only for “select customers.” Every launch post wins its own chart.