Post #2676141
2026-04-23 16:18 UTC
@redstrate@mastoart.social @jsstaedtler@mastodon.art
Many crawlers ignore this in my experience, especially the AI ones
Replies (2)
-
@redstrate@mastoart.social 2026-04-23 16:23
@rubenwardy@hachyderm.io @jsstaedtler@mastodon.art it would at least help with the legitimate ones!
-
@oblomov@sociale.network 2026-04-23 17:02
@rubenwardy@hachyderm.io @redstrate@mastoart.social @jsstaedtler@mastodon.art ironically, crawlers typically respect robots.txt (there are exceptions) but scrapers (which are the real problem) don't.