“Scraper attacks come from a huge number of sources across the net…each hit is meant to look like just another human with a web browser.”
AI companies are strip-mining the open web and dressing up every scrape to look like a human with a browser, because admitting the real scale would get them blocked. Site operators are burning engineering hours on defenses nobody asked to build, just to keep servers upright against bots harvesting training data for free. Corbet’s right that this ends one of two ways: sites lock down behind logins, or someone forces the labs to follow basic scraping etiquette. Right now it’s the former, and the open web pays for it.