Post #2972700
2024-12-28 12:52 UTC
@khobochka@mastodon.social I rescued an old forum whose administrator decided they weren’t interested anymore. I was aghast at the traffic coming from the big AI companies.
The first few weeks, I spent an hour, every day, blocking huge netblocks at the firewall. I wrote a script that summarized the heavy-hitters to the web server, investigated each one manually, then added it to the firewall’s blocklist. At the end of the first month, I was blocking nearly 1% of all IPv4 addresses.
Replies (1)
-
@denschub@mastodon.schub.social 2024-12-30 10:24
@JustinDerrick@mstdn.ca @khobochka@mastodon.social this matches my experience, yeah. I'm also convinced those companies have dedicated crawlers for specific applications like mediawiki or phpbb. I haven't seen the same aggressive behavior on any other web thing I host, and the coverage and approach to crawling the wiki almost makes me believe this is a mediawiki-specific crawler.