↓ Skip to main content

“Many models struggled to recall Obadiah 1:16 NIV word for word, often mixing up the words with other translations.”

Benjamin Kaiserbenkaiser.dev ↗

Every model got John 3:16 right except Llama 3.3 70B, which slipped in “only begotten son” from another translation. On an obscure verse only Llama 405B, GPT 4o and Claude 3.5 Sonnet passed. The smaller models blend translations and present the mix as scripture. A tool that sounds certain and quotes wrong is the worst possible tool for a text where every word matters.