“Tasks that seemed straightforward often took days rather than hours, with Devin getting stuck in technical dead-ends or producing overly complex, unusable solutions,” the researchers explain in their report.
Devin finished 3 of 20 tasks. It costs $500 a month and launched as the first AI software engineer. Fourteen tasks failed outright. The marketing shipped. The engineer did not.