Post #162095
2026-01-28 17:38 UTC
Replies (17)
-
@darkoneko@shelter.moe 2026-01-28 18:03
@osm_tech oooooooof
-
@michel42@norden.social 2026-01-28 18:15
@osm_tech do you want to share this IP-List?
-
@boz@mastodon.uno 2026-01-28 18:49
@osm_tech Please survive! I hope many people in the world will sue these aggressive companies for being the clear cause of worldwide service interruptions and quality of service disruptions and increment in electric bills, server bills bandwidth bills, RAM increase needs, etc. - they are literally killing the web.
-
@HunterZ@mastodon.sdf.org 2026-01-28 19:09
@osm_tech does coming from residential IPs mean that someone has baked a scraper into some popular tool that people don't realize is doing that?
-
@utf_7@mastodon.social 2026-01-28 19:15
@osm_tech how can you even scrape a mapsite? isnt osm just a big canvas like google maps that is also nearly impossible to automate?
-
@JonSaenzAgirre@mastodon.eus 2026-01-28 20:16
@osm_tech question. Why do people scrape server which make the data freely available? And, probably, better structured in the final product. I don't see the point.
-
@zymurgic@mastodon.online 2026-01-28 21:36
@osm_tech I wonder if the culprit will ever come forward, apologise, and change their ways? Someone tasked these proxy scrapers with ridiculous requests. Have they been targeting the main OSM API, the website interface designed for humans, or Overpass?
-
@HLunke@darmstadt.social 2026-01-28 21:47
@osm_tech Might be a good idea to become OSMF Member now or just donate some money. Membership is starting at 15£/yer https://supporting.openstreetmap.org/
-
@tykayn@mastodon.cipherbliss.com 2026-01-28 23:39
@osm_tech Have you tried some #iocaine, #anubis, and shared block lists for #fail2ban ? I think some defense guides for admin sys would be useful for a few people around here that are hosting things for osm too #fuckiascraping
-
@burtyb@widget.uk 2026-01-29 00:08
@osm_tech sounds familiar, last year I braved turning cloudflares "under attack" mode off for https://dnshistory.org/ and saw an extra 5 million requests/day (500k unique IPs) overloading things. It's still blocking >700k requests/day a month later...
-
@ClariNerd@mastodon.social 2026-01-29 00:11
@osm_tech and we can tell the scrapers are AI built because a cursory glance at the documentation on the "coders" part would've prevented this problem.
-
@apnoe_soeren@mastodon.social 2026-01-29 05:36
@osm_tech Limit the speed to Modem 14400 speed each IP for a month or so. 😅
-
@OndrejZizka@witter.cz 2026-01-29 09:23
@osm_tech 🤦♂️
-
@j@mastodon.n41.lat 2026-01-29 13:34
@osm_tech Are you able to share if the user agent of these bots has some commonality across requests? Because this looks like what I have started experiencing recently with my small, personal Mastodon instance: https://mastodon.n41.lat/@j/115966451464875347
-
@algernon@come-from.mad-scientist.club 2026-01-29 21:54
@osm_tech FWIW, if the robots attacking you are anything like the ones attacking me (~790k unique IP addresses at peak, about half of them residential), you can mitigate a large majority of them with three ifs in a trenchcoat, implementable in nginx and Caddy, and probably apache and others too. I'm able to mitigate ~60 million requests / day on a €4/month VPS. I had to scale that up once during a larger wave to a €11/month VPS, and that barely blinked when I was hit by a 2.5k request/sec wave that lasted ~4 days. (My bottleneck on the cheaper VPS was TLS, the defense mechanisms I employ are very lightweight.) Happy to help if you need a hand, just give me a shout.
-
@forestjohnson@social.pixie.town 2026-02-11 20:59
@osm_tech What do I do once I download the osm data? Is there a first party recipe for mapnik and mod_tile that is approachable and usable? I've seen some 3rd party docker-based efforts in this vein, with varying levels of activity and documentation, probably the best being https://github.com/Overv/openstreetmap-tile-server I haven't tried it yet but it seems like a good start. AI companies can piss off, I just want to be able to serve map tiles over http easily.
-
@gme@bofh.social 2026-01-28 20:35
@osm_tech@en.osm.town Could something like Anubis help you guys?