“The model was trained on 2,788,000 H800 GPU hours at an estimated cost of $5,576,000.”
DeepSeek v3 is the top openly licensed model and cost about $5.6 million to train. Llama 3.1 405B used 11x the GPU hours and benchmarks slightly worse. 18 organizations now beat the original GPT-4, and captioning 68,000 photos with Gemini 1.5 Flash 8B costs $1.68. US export controls were supposed to slow China down. They taught it to train on the cheap.