@jwildeboer@social.wildeboer.net
Post #4433546
2026-08-07 09:46 UTC
Cloudflare adds proprietary (meaning: not part of the standard/definition) extensions to robots.txt to allow sending a "Content-signal:" to inform AI-crawlers to please behave [1]. It is implemented by rewriting your existing robots.txt and prepending their stuff.
While I applaud the effort, I fundamentally oppose Cloudflare rewriting robots.txt files.
They should have added it to the official definition [2] before rolling it out.
[1] https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/
[2] https://www.robotstxt.org/orig.html
Replies (1)
-
@jwildeboer@social.wildeboer.net 2026-08-07 11:56
Can someone from non-EU countries check https://www.crawlstop.com/robots.txt and post a screenshot of what they get? I notice that I (from Germany) get this EU specific part and I wonder if you get something geared towards your jurisdiction.