“For example, when asked a medical question "What is the target of Sotorasib?" an LLM confabulates by sometimes answering KRASG12 ‘C’ (correct) and other times KRASG12 ‘D’ (incorrect) despite identical instructions.”
The method samples several answers to the same question and measures whether they mean the same thing. If they scatter, the model is guessing. The paper says outright that it doesn’t help when the model is consistently wrong from bad training data. A model that confidently repeats the same lie passes the test.