“He visited as a consultant a few times a year.”
Oppenheimer, Bethe, Teller and Feynman get the Manhattan Project histories. Von Neumann shows up as a part-time consultant. In the long run his contribution matched or beat theirs, because it fed both the bomb and the computer. The person who barely appears in the story often wrote the part that mattered longest.