Post #4433780

Cloudflare (>20% of all domains) has started automatically implementing DSM Dir art 4 rights reservation (ie copyright/TDM objection to scraping) for all sites that do not have a robots.txt file. They also adding purpose fields (new content-signal field with search, AI train, AI use) to robots.txt unilaterally and are enforcing it through monitoring and analysis. https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/