Post #3474262
2026-06-28 10:02 UTC
@drgeo@mamot.fr @doctormo@floss.social scrapers trying to get content for training AIs but they often don't respect site owners wishes as well as assuming everyone has the infrastructure budget of Google.
Some of them don't politely crawl the site with a slow rate of requests they bombard it causing a DOS/DDOS (overwhelming the site and taking it offline)
Replies (1)
-
@ohir@social.vivaldi.net 2026-06-28 11:01
@doctormo@floss.social @Dragon@toast.dragon2611.net > scrapers ... don't respect site owners wishes Though they do respect a magic incantations in the meta headers: ``` ``` (Tested on my small shared hosting that came cheap but with 1TB/mo limit what scrappers started to suck in less than a day.) @drgeo@mamot.fr @algernon@journa.host