Post #3474268
2026-06-28 11:30 UTC
@Dragon@toast.dragon2611.net @cartocalypse@norden.social @drgeo@mamot.fr @doctormo@floss.social And even then, if it's juicy enough, crawler makers update their crawler to solve the challenges. E.g. Anubis does not work for Codeberg anymore because crawlers can deal with it:
https://social.anoxinon.de/@Codeberg/115033790447125787
Replies (1)
-
@danieldk@mastodon.social 2026-06-28 11:34
@Dragon@toast.dragon2611.net @cartocalypse@norden.social @drgeo@mamot.fr @doctormo@floss.social I fear that in the end only something like Cloudflare works, since they can see a large chunk of internet traffic and therefore detect crawlers across many different sites. This is necessary because these residential crawlers often only do one or a few requests to each site, so it is hard to detect them from a single site. We need a non-profit CDN!