Post #1584732
2026-04-23 16:09 UTC
Replies (5)
-
@portaloffreedom@social.linux.pizza 2026-04-23 16:45
@jsstaedtler@mastodon.art if you want to get evil, create a zip bomb or one those infinite generating bulllshit website so the evil crawlers get stuck there.
-
@redstrate@mastoart.social 2026-04-23 16:11
@jsstaedtler@mastodon.art a dumb solution would be to tell robots to not index the page (robots meta tag) if there is any tag queries, which i assume you can do via PHP. edit: or if you want individual tags indexed, at least reject robots for queries of more than one tag?
-
@chronocide@sunny.garden 2026-04-23 17:07
@jsstaedtler@mastodon.art I've have / had this issue at work and to my surprise even Google's own bots (the not awful ones) are exceptionally stupid at this, even to the point where Google Search Console (SEO management tool) tells me to stop blocking nonsense query parameters. Like, if I tell you that search result pages shouldn't be indexed because there's a gazillion of them, please don't STILL crawl them and tell me off for not allowing you to index them!
-
@jeff@higgy.social 2026-04-23 20:25
@jsstaedtler@mastodon.art Just thinking out loud, but you could maybe catch the user agent of known crawlers in the PHP and send them a different page. Won't work for all crawlers, but would likely stop a chunk of them.
-
@pentup@mastodon.art 2026-04-24 03:54
@jsstaedtler@mastodon.art My thought was very similar, but rather than trying to figure out which tags are exclusive, just don't let the user add filters if there are 0 or 1 results, because why would they? (Perhaps even if there's only one page of results?)