Post #2676144
2026-04-23 17:02 UTC
@rubenwardy@hachyderm.io @redstrate@mastoart.social @jsstaedtler@mastodon.art ironically, crawlers typically respect robots.txt (there are exceptions) but scrapers (which are the real problem) don't.
Replies (1)
-
@rubenwardy@hachyderm.io 2026-04-23 17:24
@oblomov@sociale.network @redstrate@mastoart.social @jsstaedtler@mastodon.art Yeah. Crawlers also use a sensible user agent whereas scrapers pretend to be actual users