Post #1584738

2026-04-23 15:29 UTC

I have an obnoxious problem with crawlers eating bandwidth on my personal web site—not just the fact that crawlers consume so much bandwidth, but rather a behaviour that is absolutely next-level. And I think it's something that precludes the use of caching, but there are probably many of you with more knowledge than I have and who may know what can be done 🧵⤵️ #IndieWeb #WebDev #PersonalSite

Replies (7)

@jsstaedtler@mastodon.art 2026-04-23 15:38

My site uses PHP to produce HTML output—no flat HTML files. There is an image gallery, and the images have tags. It's here if you actually want to see it: https://bigraccoon.ca/gallery When you filter on a tag, it adds a parameter to the URL, e.g. "domain[dot]com?tag=2026". That loads the gallery, but only displays images tagged with "2026". You can filter further on more tags, e.g. "?tag=2026%2CPencil" ("%2C" is a URL-encoded comma), which would show images from 2026 drawn in pencil. 🧵2/?

Open ##1584737
@vga256@mastodon.tomodori.net 2026-04-23 15:33

@jsstaedtler@mastodon.art I can't remember - are you self-hosting or using a paid host?

Open ##2676128
@jsstaedtler@mastodon.art 2026-04-23 17:04

@tkissing@mastodon.social @oblomov@sociale.network That project looks very promising actually, thank you!

Open ##2676139
@rubenwardy@hachyderm.io 2026-04-23 15:41

@jsstaedtler@mastodon.art The exact thing has happened to me recently with the tags. I now require users to log in to filter by multiple tags and I've blocked the subnets of the bots If I wanted to allow guest users to search by multiple tags, I'd probably try the following options - (1) changing it to a POST request (2) requiring JavaScript (3) using Anubis (4) looking into ip masked rate limiting, so a rate limit for like multiple ip addresses in the same block I wrote a blog post about my situation here https://blog.rubenwardy.com/2026/04/16/contentdb-ddos/

Open ##2676161
@cb@boop.bleepbop.space 2026-04-23 15:50

@jsstaedtler@mastodon.art I've been using Iocaine, which is specifically intended to mess with AI bots, but it can also help with "normal" bots too https://iocaine.madhouse-project.org/ of course that still eats up some of your server's power. I work for a web hosting company and frequently we'll just make a list of "bad bots" in an .htaccess file to block them. The server still has to reply to their requests but doesn't have to serve them any real data

Open ##2676164
@jsstaedtler@mastodon.art 2026-04-24 13:57

@CarbonCatalyst@mastodon.social I'm on a shared webhost, so not much access to the backend outside of Apache

Open ##2676166
@Numerfolt@kirche.social 2026-04-23 20:34

@jsstaedtler@mastodon.art Maybe Anubis could be worth a look? https://github.com/TecharoHQ/anubis Havent tried it myself though... And then there's an offensive measure: Iocaine https://iocaine.madhouse-project.org/ I fear it does also kill site indexing, but it feeds the Crawlers an endless amount of bullshit.

Open ##2676167