Post #1020559

2025-10-25 16:06 UTC

We apologize for the long performance degradation today. Finally, we identified all of the &#39;tricks&#39; that AI crawlers found today. They no longer bypass the anubis proof of work challenges. A novelty for us was that AI crawlers seem to not only crawl URLs that are actually presented to them by our frontend, but they converted the URLs into a format that bypassed our filter rules. By the way, you can track the changes we have been doing via https://codeberg.org/Codeberg-Infrastructure/scripted-configuration/compare/51618~1..e4aac

Replies (15)

@lax@tooot.im 2025-10-25 16:08

@Codeberg wow it's so much fasterier

Open ##1359201
@mason@partychickens.net 2025-10-25 16:13

@Codeberg They're malicious.

Open ##1359202
@AmyZenunim@unstable.systems 2025-10-25 16:16

@Codeberg 🫡

Open ##1359204
@bit101@mstdn.social 2025-10-25 16:19

@Codeberg Ah I did notice a super long push time this morning. That explains it. Sucks that we have to deal with that crap. But thanks for the transparency.

Open ##1359205
@clementd@framapiaf.org 2025-10-25 16:43

@Codeberg AI is malware

Open ##1359206
@corpsmoderne@mamot.fr 2025-10-25 16:44

@Codeberg fucking parasites.

Open ##1359207
@Fedihacker@masto.es 2025-10-25 17:01

@Codeberg Malware has the same behaviour.

Open ##1359208
@gimulnautti@mastodon.green 2025-10-25 17:16

@Codeberg Now OpenAI is giving real users their Atlas browser, so they can scrape while users bypass the security and provide them with logins. Disguisting.

Open ##1359209
@indyradio@kafeneio.social 2025-10-25 17:16

@Codeberg I'm going to try a normal, respectful, old fashioned crawl and see if that's possible. This is the search engine for indyradio, it has documents google won't find and for me usually works better. excuse the plug. I'll let you know what results I got. https://lookdown.org

Open ##1359212
@bjarne@social.bvoigtlaender.de 2025-10-25 17:35

@Codeberg Was it only today? I felt like it has been slow for the past few weeks. Its really fast now though! Thank you, once again, for the hard work.

Open ##1359213
@soop@outerheaven.club 2025-10-25 17:48

@Codeberg do you really have to be using anubis and not go-away or iocaine cuz anubis is kinda uncomfy with its corporateness

Open ##1359214
@amte@mastodon.world 2025-10-25 18:21

@Codeberg Already witnessing that the harm and impact of AI for all-day life is way beyond it's benefits!

Open ##1359218
@aerofreak@hessen.social 2025-10-25 19:22

@Codeberg Now give this thread some hashtags because it's important.

Open ##1359219
@claudius@darmstadt.social 2025-10-25 21:11

@Codeberg AI companies crawl our websites. We ask that they stop by using the industry standard robots.txt AI companies ignore those rules. We start blocking the companies themselves with conventional tools like IP rules. AI companies start working around those blocks. We invent ways to specifically make life harder for their crawlers (stuff like Anubis). AI companies put considerable resources into circumventing that, too. This industry seriously needs to implode. Fast.

Open ##1359220
@2something@transfem.social 2025-10-25 20:16

@Codeberg@social.anoxinon.de Are crawlers that fail the test getting sent to a maze of procedurally-generated junk, or are they just told they failed?

Open ##2909791