“As a thought exercise, in 2021, DoorDash ML models made 10 billion predictions a day. If each prediction costs $0.004, that’d be $40 million a day!”
A demo needs one good answer. Production needs ten billion a day at a price that doesn’t bankrupt you. Huyen measured gpt-3.5-turbo at around 500ms for a one-token reply, and OpenAI offers no SLA. Teams tell her they redo the build versus buy math every week. The model is the easy part. The invoice and the latency are the product.