Post #1347750

2025-04-20 21:35 UTC

I have been trying for hours to debug what I thought was a #Treafik regression causing massive CPU usage to the point that my own (very small) dockerized services I host for my family are getting slow and hard to reach. But no, my meager virtual server is just being DDoSed by stupid #AI bots downloading pieces of my webpage over and over and over again. It looks like the steps described in https://www.mayrhofer.eu.org/post/defenses-against-abusive-ai-scrapers/ are no longer working, and I need to start looking into actual IP blocking. However, as AI fraudsters are resorting to using massive client pools for downloading, that will also become difficult. This is new. The AI scraper bots amounting to over 90% of all traffic until a couple of weeks ago were annoying, but my services still worked. Now it has reached the level of active denial of service.

Replies (6)

@rene_mobile@infosec.exchange 2025-04-21 15:55

I declare partial success with much more aggressive connection, request, and transfer rate throttling in the embedded #nginx instance that serves my static page (plus the dynamic link maze that caught the stupid "AI" scraper bots...) behind #traefik (causing the TLS termination part to be overloaded and blocking authenticated users from legitimate access). https://www.mayrhofer.eu.org/post/defenses-against-abusive-ai-scrapers/nginx-default.conf is the current rate limiting config, https://www.mayrhofer.eu.org/post/defenses-against-abusive-ai-scrapers has the explanations. Something like https://blog.lrvt.de/configuring-crowdsec-with-traefik/ will probably have to be the next level of escalation to deal with the issue on a global level.

Open ##2409437
@newstik@social.heise.de 2025-04-21 02:07

@rene_mobile@infosec.exchange I see a class action lawsuit... :-(

Open ##2409439
@arouene@mastodon.social 2025-04-21 07:45

@rene_mobile@infosec.exchange someone has created a proxy to fight the AI bots : https://anubis.techaro.lol/

Open ##2409440
@hackbyte@friendica.utzer.de 2025-04-21 09:59

@rene_mobile@infosec.exchange In the end, we may need ip-range traffic-shaping stuff to just slow all users down to _user_ level usage. 🙁

Open ##2409442
@gunstick@mastodon.opencloud.lu 2025-04-21 19:00

@rene_mobile@infosec.exchange To make the AI bot life miserable, install anubis proof of work https://anubis.techaro.lol/ Or checkpoint cryptographic challenge https://github.com/vaxerski/checkpoint Or nepenthes infinite maze https://zadzmo.org/code/nepenthes/ Or quixotic nonsense generator https://marcusb.org/hacks/quixotic.html

Open ##2409443
@rnbwdsh@chaos.social 2025-04-23 12:29

@rene_mobile@infosec.exchange i'm curious: do they at least have a proper user agent, or just chrome as it's probably a puppeteer/selenium thing? My suggestion: You embed a script that gets blocked by easylist - and anyone who hits it gets blocklisted for 10 minutes + forwarded to google via js. I'd asume even jsblocking crawlers would still try to open the js - unlick jsblockers or adblockers.

Open ##2409445