Post #2772605
2026-04-18 16:39 UTC
@nathanu@lounge.town I don't suppose you could detail exactly what these anti-AI directives are in your robots.txt so we can be sure to avoid them to increase our speed? ;)
Replies (1)
-
@nathanu@lounge.town 2026-04-18 16:46
@phoikoi@hachyderm.io So, it was specifically the last two here that it complained about: https://nathanupchurch.com/robots.txt. I always forget to bookmark the resource I use to get that monster of a file, but I have a feeling it might have been this: https://robotstxt.com/ai.