Post #2676146
2026-04-23 16:25 UTC
@redstrate@mastoart.social Ah, this sounds promising! I don't want to make my site invisible on the greater Web by blocking all bot crawlers, but I'd be fine with them only loading URLs with no queries/parameters (anything after a ?). I'll look into that meta tag, though I acknowledge the other reply here that bots can happily ignore that.
Replies (1)
-
@gemelen@mammut.moe 2026-04-23 16:32
@jsstaedtler@mastodon.art @redstrate@mastoart.social The problem is that lots of crawlers do not respect robots.txt (especially those run by "AI" companies). Thus people go for other solutions, to make it too expensive on the side of the crawler, like iocaine - https://firesphere.dev/articles/iocaine-the-deadliest-poison-known-to-ai, or anubis - https://anubis.techaro.lol