Post #1973768
2026-04-29 10:13 UTC
@kirb I am seeing similar behaviour on smaller project's websites. There is 1 aspect that I see rarely mentioned: The scraper bots cannot speak proper HTTP at all. They keep the TCP connection alive after the first request, and never re-use the connection for a second request. requiring the server to timeout, which quickly causes resource exhaustion and running into connection limits on smaller sites, even when the server would otherwise be perfectly capable of handling the CPU load.
Replies (1)
-
@manx@mastodon.online 2026-04-29 10:17
@kirb Once they have already scraped millions of URLs and put them in their todo queue, a captcha is not even a viable solution any more, because their hostile use of TCP connections is unaffected by the captcha. In some cases, I had to temporarily or permanently blackhole whole /8 subnets.