Post #2686065

@tdp_org@mastodon.social @SimmerVigor@mastodon.social Not being well-mannered is definitely a problem. One of my sites got hammered by an AI crawler a while ago, we found them via the user agent and they have actually published their source code. The tool has an explicit option to turn off respecting robots.txt. When I e-mailed them about it, they did not understand why that might be a problem 🙄