↓ Skip to main content

“It’s futile to block AI crawler bots because they lie, change their user agent, use residential IP addresses as proxies, and more.”

Xe Iasoxeiaso.net ↗

robots.txt told every bot to stay out. Amazon’s crawler kept hammering the git server anyway. Blocking the user agent failed because about 10% of requests did not use it. Amazon gets the training data and the small host pays for the bandwidth.