Post #3502223

@tyler@typing.ink @aphyr@woof.group The reason iocaine serves garbage is that it can serve poisoned (and later recognizable) URLs, to overload the crawler's queue. This way, when they come back riding remote controlled chromes, they'll hit poisoned URLs, and will end up in an infinite maze of garbage too. It's also possible to configure iocaine to block certain requests (temporarily firewalling the IP off for a time) - for example, when they hit a poisoned URL. This firewalling bit got my incoming dily requests down from ~100m/day to ~2.8m/day. So iocaine is perfectly capable of blocking these. But blocking them prevents poisoning the crawling queue, which reduces the effectiveness of poisoned URLs, so by default, poison's served. Said poison is also extremely cheap to generate (and will be even cheaper in the not too distant future), cheaper than serving a small static file from a filesystem even. Every crawler trapped in the maze is a crawler that isn't hitting your backend. With that said, I see two major differences between Anubis and iocaine: Anubis - usually - presents a proof of work, which both crawlers and humans need to bypass. Crawlers have more money to burn, and if they really want to, can bypass it (like they did against Codeberg). If a proof of work is presented, a crawler has a chance to get through. iocaine is passive, and uses various heuristics to catch crawlers. If it traps a bot, there is no way out. There's also no proof of work challenge for the human visitor, either. They both aim to do the same thing: reduce the load on services behind them. Their approach is wildly different, even if the goal's similar.