“In many ways, this feels less like engineering and more like a search for spells.”
gpt-4o reached about 1540 Elo, but only with three in-context examples and a trick that makes it repeat the whole move list before answering. The same recipe played poorly on gpt-4. The best explanation on offer is that OpenAI trained on curated chess games and the chat tuning buried the skill, and nobody outside OpenAI can test that. Getting good output from a frontier model is still incantation, not engineering.