“We’re releasing an open source system, based on FSDP and QLoRA, that can train a 70b model on two 24GB GPUs.”
Four H100s with 320GB of RAM cost around $150,000. Two gaming cards with 24GB each cost under $10,000 and run at similar speed. The only gap was memory, and the big labs had no reason to close it because they already bought the expensive hardware. NVIDIA’s whole data center markup rests on 56GB of RAM.