Major AI Players Adopt Unified Opt‑Out Standard for Web Scraping

OpenAI, Google, Microsoft and Anthropic announced they will honor a new “policy token” proposed by Anthropic’s Kevin Kelley within the Internet Engineering Task Force. The token lives in a site’s robots.txt file and signals the publisher’s AI‑training preferences to every participating crawler at once.

Instead of configuring separate opt‑out rules for each vendor, a publisher adds one line to robots.txt and the signal is respected across all four companies. Microsoft will also expose finer‑grained controls, letting sites opt out of model training while still allowing grounding — the practice of pointing a model to verified sources to curb hallucinations.

Anthropic has already implemented the token; Google and Microsoft plan to add support next year, and OpenAI expects to roll it out in the coming months.

Why it matters for GPU and AI infrastructure

Unauthorized scraping forces GPU clusters to ingest and process massive, unlicensed datasets, inflating compute costs and legal risk. A universal opt‑out reduces unwanted traffic, lets data owners monetize or restrict access, and helps AI operators allocate GPU capacity to vetted, high‑value workloads.