“At the beginning of July, its robots.txt file was updated to block web crawlers it doesn’t have agreements with.”
Reddit now shows up in Google, which pays for the data, and not in Bing, which doesn’t. Steve Huffman calls Microsoft, Anthropic and Perplexity freeloaders for using posts that Reddit’s users wrote for free. Those users get nothing from the Google or OpenAI deals either. Reddit didn’t close the open web. It put up a toll booth and kept the fare.