“Claude 2.1 has also made significant gains in honesty, with a 2x decrease in false statements compared to our previous Claude 2.0 model.”
Anthropic doubles the context window to 200,000 tokens, about 500 pages. The full window is reserved for paying Pro users. A 2x drop in false statements is also a confession about how often 2.0 made things up. The benchmark is Anthropic’s own curated question set, graded by Anthropic.