Post #1559066

2026-01-27 21:18 UTC

@osm_tech I wonder if there's a way to fail2ban requests coming in faster than typically found in human requests.

Replies (3)

@osm_tech@en.osm.town 2026-01-27 22:44

@BalooUriza We use fail2ban to handle some of this with custom rules, but eventually fail2ban becomes a bottleneck after 100,000 IP addresses.

Open ##1559067
@ThatPrilla@theatl.social 2026-01-28 03:46

@BalooUriza @osm_tech Cycling to new IPs is trivial, I ban a few thousand IPs and cidr ranges in my WAF, I’ll see 75% of them show up the next time the scraper hits. Then after that most don’t show up again and the next scrape comes from a mostly new set of IPs. I’ve see A few instances where they will cycle IPs during the same scraping event if some of them are blocked. I’ve got scrapers that will send every request from a unique IP. There is a lot of money to be made right now offering hard to block scraping services or tools to enable them.

Open ##1919668
@nicd@masto.ahlcode.fi 2026-01-28 07:19

@BalooUriza The problem is, who do you ban? Since the requests keep changing IPs and user agents.

Open ##1919669