AI Crawling Is Becoming a Policy Layer, Not Just a robots.txt Setting
The important change is not another AI crawler blocklist. Search, model training, and user-directed agents are increasingly being treated as different forms of web access with different permissions.
2026-09-19