Post #2413513
2025-04-04 07:03 UTC
@lavaeolus@fedihum.org @SafeguardingResearch@fedihum.org Just a note that when your set your crawler to scraping a gov site, be reasonable about how frequently your requests are posted. The AI/MML bots are out there doing the same damn thing, and some sites have set their defenses on DROP if they don't like the look of your jib. Also, if your IP traces back to Malaysia or Lithuania or some other out-of-the-way non-US place, you'll look even more suspicious.
Replies (1)
-
@lavaeolus@fedihum.org 2025-04-04 07:05
@roadskater@mastodon.social yes, this is an overall issue - fighting with 403s is part of the daily work re bots: https://go-to-hellman.blogspot.com/2025/03/ai-bots-are-destroying-open-access.html?m=1 @SafeguardingResearch@fedihum.org