“On math word problems GPT4 with CoT managed to solve 12.5% of all problems, but GPT4 and Prolog solved 100%.”
The 100% comes from a new dataset built to measure this exact method. On GSM8k, the benchmark everyone else uses, Prolog added 3% for GPT4. The model translates the problem into rules and a Prolog interpreter does the actual reasoning. The gain belongs to the interpreter, not the model.