← Feed @0x4261756D@mastodon.online Post #2725954 2025-01-15 13:04 UTC @aaron@chirp.zadzmo.org Do LLM crawlers typically honour robots.txt? Otherwise this could be a way to avoid spare innocent crawlers. Replies (0) No replies.