Post #2420303
2025-05-12 09:50 UTC
Replies (26)
-
@gsuberland@chaos.social 2025-05-12 10:01
@tdp_org@mastodon.social hot damn, that's a big difference
-
@q@glauca.space 2025-05-12 10:03
@tdp_org@mastodon.social do you do anything different based on if a bot is known or not?
-
@kfh@chaos.social 2025-05-12 10:11
@tdp_org@mastodon.social does the filter just match known crawler user agents against their known origin networks?
-
@ewenmcneill@cloudisland.nz 2025-05-12 10:13
@tdp_org@mastodon.social @gsuberland@chaos.social that’s entirely consistent with what I’ve seen recently. Both Google and Meta user agents seem to be deliberately misused by very inconsiderate crawlers coming from Definitely Not The Right IP Space For The Claimed Agent. I suspect tying those user agents to known IPs is going to become essential soon :-/
-
@slink@fosstodon.org 2025-05-12 10:18
@tdp_org@mastodon.social i assume you are aware that also meta provides official guidance on how to identify their bots? https://developers.facebook.com/docs/sharing/webmasters/web-crawlers
-
@dngrs@chaos.social 2025-05-12 10:37
@davidgerard@circumstances.run ⬆️
-
@ross@hachyderm.io 2025-05-12 10:58
@tdp_org@mastodon.social Next step: immediate IP block. I have strong feelings towards the companies that sell access to residential IPs across the planet as proxies for crawlers.
-
@tychotithonus@infosec.exchange 2025-05-12 11:31
@tdp_org@mastodon.social Fantastic! Are you at liberty to say how your classifier(s) are implemented? In my ideal world, Apache / nginx / Caddy / etc would have milter-like "call out to this code with this metadata and receive one of N responses" capability, so that even small sites could do things like this.
-
@davidgerard@circumstances.run 2025-05-12 11:31
@tdp_org@mastodon.social i won't describe the mitigation we applied on rationalwiki, but we saw even more drastic and successful results
-
@cehteh@karlsruhe-social.de 2025-05-12 11:47
@tdp_org@mastodon.social I can see this escalating, next is that they use semi random user-agent strings and/or don't publish themself. For myself (small open source project server) I just made the simplest behavioral analysis possible: Firewall limits new connections from an IP to a slow rate (after some initial burst). Normal user clients would use pipelining, they wont make hundreds of connection attempts. Webserver gives the first few MB for free and then limits the bandwidth to a crawl.
-
@kasperd@westergaard.social 2025-05-12 11:53
@tdp_org@mastodon.social This is interesting information and certainly something I will consider making use of myself. All legitimate crawlers should make it easy to verify the authenticity of the requests they make. I know that for Googlebot the recommended way to verify the authenticity of the request is as follows: Do a reverse lookup of the IP address.Verify that the resulting name is under the googlebot.com domain.Do a forward lookup of the name to verify you get the original IP address back. I don’t know if similar steps have been published for other legitimate crawlers.
-
@1a1nC@mastodon.online 2025-05-12 12:17
@tdp_org@mastodon.social nice.
-
@SimmerVigor@mastodon.social 2025-05-12 12:37
@tdp_org@mastodon.social take a look at https://datatracker.ietf.org/doc/html/draft-meunier-web-bot-auth-architecture-01
-
@jalict@mastodon.gamedev.place 2025-05-12 12:42
@tdp_org@mastodon.social Is the bottom line zero or is it some other fixed number? Looks crazy high. Also would this break Facebook Embeds?
-
@gimulnautti@mastodon.green 2025-05-12 13:30
@tdp_org@mastodon.social So crawlers definitely not identifying themselves appropriately, if I get your point correctly? In other words, these crawlers could be anybody, and are lying about their origin?
-
@csolisr@hub.azkware.net 2025-05-12 13:38
@tdp_org@mastodon.social What tools do you use for ASN validation by the way?
-
@tauon@possum.city 2025-05-12 13:52
@tdp_org@mastodon.social i only run my small personal website but i was planning on doing a similar mitigation, i'm glad to see that it works
-
@kuzko@kuzko.eu 2025-05-12 14:22
@tdp_org@mastodon.social that's a writeup a lot of people would be interested in I think.
-
@bertkoor@mastodon.social 2025-05-12 15:12
@tdp_org@mastodon.social What I don't understand is, I had expected the improved detection makes the "known bot" graph go up with your change. How come the numbers go down? You did not scare them away, did you?
-
@BubblegumYeti@mastodon.social 2025-05-12 18:51
@tdp_org@mastodon.social Akamai blocks this for us
-
@mark@mastodon.fixermark.com 2025-05-12 20:35
@tdp_org@mastodon.social Well done! I know Google did / does something similar with what were actually "legitimate" Google spiders to avoid cloaking; knowing that sites would show different output to a Google crawler vs. other clients, they'd intentionally spot-check some sites with non-Google-identifying crawlers intended to look like regular users to root out scams. But I sincerely doubt that's what you were seeing with those "Meta" crawlers on non-Meta IPs (and, honestly, if you knocked out Meta's trust-but-verify while administering your own site... That's on them, not you).
-
@hobs@mstdn.social 2025-05-12 20:37
@tdp_org@mastodon.social Meta could be using residential proxies. Most scrapers (and bot nets) do.
-
@philip@mastodon.mallegolhansen.com 2025-05-12 23:02
@tdp_org@mastodon.social Thanks for sharing, interesting to see the impact.
-
@altf4@hostux.social 2025-05-12 23:10
@tdp_org@mastodon.social So much bot traffic is starting to get really demotivating for hosting stuff
-
@froztbyte@mastodon.social 2025-05-13 13:15
@tdp_org@mastodon.social nice, how are you implementing it? RIR dumps in some format with some kind of accelerated lookup (guessing by prefix a la routing trie or similar)? (off the top of my head I dunno if any popular open source servers have this natively, haven’t had to look for a while, but now I feel like gonna go check all the usual suspects ;D)
-
@DrHyde@fosstodon.org 2025-05-15 07:47
@tdp_org@mastodon.social I knew that the code I wrote for Auntie back in 2000-ish so we could make sure we only streamed #RealAudio of the Olympics to people in EBU countries would be useful again one day! It did geolocation by AS.