Post #1584734

2026-04-23 15:56 UTC

So it loads my gallery page, and sees the list of tags: maybe 50 different links, all of which load the gallery page with a new filter applied. So it loads one, like "?tag=2026". On the resulting page, there are still 50-odd tag links available. So it loads another one, and the URL now includes "?tag=2026%2C2025". Which is nonsense, but the page still loads. Well, there are 0 images to show on that page, but still more tags to open! So next the bot opens "?tag=2026%2C2025%2C2024"... 🧵5/?

Replies (2)

@jsstaedtler@mastodon.art 2026-04-23 16:03

How many permutations of tags are there? A butttonne, and the bot will diligently check out ALL OF THEM. Thousands and thousands of page loads! And even though all of them have 0 images to display, there will still be a tag list to choose from, and it will always visually update to indicate which tags are currently selected. So the page can't just be saved in a static HTML file, and the bot isn't going to load anything from it's own cache. 🧵6/?

Open ##1584733
@oblomov@sociale.network 2026-04-23 15:59

@jsstaedtler@mastodon.art an easy way to catch this is that these scrapers generally don't send Referer headers, so you can kill these by checking that a valid Referer header is present in tag search. This will have false positives for humans that try to be too smart though.

Open ##2676136