“If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really).”
The diaspora project’s sites took 11.3 million requests in 60 days, and GPTBot alone made 24.6% of them. Amazonbot added another 14.9%. Googlebot made 0.14%. The AI crawlers ignore robots.txt, come back every six hours and crawled every diff in the wiki’s edit history. Volunteer projects pay the hosting bill so OpenAI and Amazon can train on them for free.