“We are experiencing dozens of brief outages per week, and I have to review our mitigations several times per day to keep that number from getting any higher.”
LLM crawlers ignore robots.txt and hammer SourceHut’s most expensive pages from tens of thousands of residential IPs. DeVault spends up to his whole week fighting them. The mitigations block real users too. AI companies push their costs onto projects that run on less than one Google engineer’s salary.