Post #2676147

@jsstaedtler@mastodon.art @redstrate@mastoart.social The problem is that lots of crawlers do not respect robots.txt (especially those run by "AI" companies). Thus people go for other solutions, to make it too expensive on the side of the crawler, like iocaine - https://firesphere.dev/articles/iocaine-the-deadliest-poison-known-to-ai, or anubis - https://anubis.techaro.lol