Post #2720204
2026-01-13 09:58 UTC
All crawlers ignore robots text since the very start. Anyway, if THAT is the problem then IT is a problem, not the LLMs as a whole.
Replies (2)
-
@FauxLiving@lemmy.world 2026-01-13 17:16
You can tell when you’re talking with someone who has been given the position of ‘AI Bad’, but doesn’t actually understand the moral positions or technological details that form the foundation of that argument by how confidently they repeat some detail that is clearly nonsense to anybody with knowledge of the subject.
-
@arcterus@piefed.blahaj.zone 2026-01-15 19:46
If this were true (which is nearly impossible since you said “all"), stuff like Anubis wouldn’t exist since you could just toss up a crowd-sourced robots.txt and be done with it.