Among the same 143 large websites tracked every month since January 2023, the share whose robots.txt tells Anthropic’s ClaudeBot to stay out of the whole site rose from 4% to 26% by October 2026, while the share blocking OpenAI’s GPTBot peaked at 26% from December 2023 to February 2024 and has since settled near 20%, as most tracked publishers that announced OpenAI deals let GPTBot back in. That split is the backdrop to the shared AI-training opt-out token that OpenAI, Google, Microsoft and Anthropic have agreed to honor, as SI News reported: today sites treat each lab’s crawler differently, and one rule would apply to all four at once.
Sources & Notes
- Original story: Major AI Labs Back Shared Training Opt-Out | Google Satellite Runs Gemma 4 in Orbit | Claude Agent Performs Mouse Brain Injection
- Data sources: The Walled Web (n.d.). Is AI getting blocked more over time? Monthly robots.txt blocking of AI training bots, January 2023 to November 2026
- Other references: The Walled Web (n.d.). Who blocks GPTBot?; Calvano, P (2025). AI Bots and Robots.txt; PPC Land (2026). Google, Microsoft, OpenAI and Anthropic back one AI training opt-out
- Notes: Each point is the share of a fixed cohort of 143 large websites whose robots.txt disallows the whole site for that bot in that month, as published (rounded to whole percent) by The Walled Web, which reconstructs months before its own daily checks from Internet Archive copies of each robots.txt file. The 4% baseline before GPTBot launched in August 2023 is sites that already blocked every bot they do not name. Months shown run from 2023-01 to 2026-10; the partial November 2026 month is excluded. Over the same months, Google-Extended rose from 4% to 20% and Common Crawl’s CCBot from 4% to 25% (not plotted). The Walled Web reports that nine of the ten tracked sites whose owners announced OpenAI agreements let GPTBot back in within about three months. robots.txt is a voluntary request and blocking does not undo past collection; a 2026-11-01 spot check of nytimes.com, wsj.com and theatlantic.com robots.txt files matched the tracker. Data accessed 2026-11-01.