IA · 1 November 2026 · 4 min read
One signal to stop every crawler: AI giants back shared scraping opt-out standard
In brief: OpenAI, Google, Microsoft, and Anthropic have committed to adopting a unified opt-out standard for artificial intelligence training scrapers. Proposed at the Internet Engineering Task Force (IETF) by Anthropic standards technical lead Kevin Kelley, the mechanism integrates a shared policy token into the standard robots.txt file, doing away with the fragmented landscape of vendor-specific crawl directives.
by Team Mocchi's
For decades, the public web relied on the quiet convention of robots.txt, a plain-text file created in 1994 to signal search engine crawlers which directories to index and which to leave untouched. With the sudden rise of generative AI and its massive appetite for training text, that vintage mechanism began to buckle under pressure. Website owners and engineering teams were forced into an endless game of whack-a-mole, configuring separate rules for dozens of proprietary crawlers — from GPTBot to Google-Extended and ClaudeBot — while worrying that strict blocks might inadvertently tank their regular search visibility.
Now, the industry's major players are moving toward a unified protocol. As reported by The Verge, OpenAI, Google, Microsoft, and Anthropic have announced they will adopt a shared standard allowing web publishers to signal their model-training preferences through a single, consistent directive.
From User-Agent clutter to a shared syntax
The initiative was submitted to the Internet Engineering Task Force (IETF) by Kevin Kelley, technical lead for standards at Anthropic. The draft outlines a shared "policy token" embedded directly within standard robots.txt syntax. Rather than crafting bespoke allow or disallow rules for every vendor's individual crawler identity, site operators can declare their overarching training permissions once, covering all participating companies.
Until now, managing bot traffic had become an administrative nightmare for infrastructure teams. Denying access to a headline bot rarely guaranteed that a company would not deploy secondary scrapers under different names to collect raw datasets, forcing webmasters to analyze server access logs constantly to identify and throttle incoming traffic.
Separating search indexing from model training
A central pain point the standard seeks to resolve is the conflict between brand discoverability and copyright protection. For smaller publishers and digital businesses, opting out of aggressive crawlers often carried the existential risk of disappearing from standard search indices on Google or Bing, destroying incoming referral traffic.
While some providers previously introduced separate User-Agents to distinguish general indexing from foundation-model training, the lack of an overarching standard created widespread confusion. The IETF specification aims to formalize this split at the internet protocol level, enabling sites to remain fully indexable for search queries without granting blanket permission for their content to be absorbed into model pre-training corpora.
An open framework after years of friction
The alignment among fierce commercial rivals arrives after two years of escalating copyright lawsuits, high-stakes licensing battles with news organizations, and pushback from open-source maintainers. Indiscriminate scraping has severely strained goodwill across the open web.
By anchoring this mechanism in the open governance of the IETF, the founding tech companies gain a transparent framework to showcase compliance and cooperation with evolving global privacy and digital market regulations. Nevertheless, voluntary standards carry inherent limitations: while OpenAI, Google, Microsoft, and Anthropic agree to honor the token, the rule cannot bind rogue scrapers, gray-market data brokers, or unregulated offshore scraping networks that ignore robots.txt entirely.
Mocchi's take
For businesses running proprietary content platforms, e-commerce stores, or SaaS technical documentation, this unified IETF standard brings much-needed clarity to day-to-day web operations, removing the friction of maintaining fragmented robots.txt rules. However, engineering leaders must keep in mind that a protocol-level signal remains an honor system among compliant actors, not an access-control firewall. Safeguarding high-value intellectual property and sensitive customer data will still require solid network-level defenses, robust authentication, and deliberate API boundaries alongside standard crawler directives.