@GroupNebula563@mastodon.social
Post #4473055
2026-08-10 02:08 UTC
@Haste@mastodon.social what i will say is that there are still good indexing bots out there, like the one belonging to https://marginalia-search.com - my general approach has been to allow indexing but disallow any AI crawlers, and send those to a honeypot. if you convince the AI crawlers that the honeypot *is* the website instead of just trying to block them off completely, they're less likely to try and evade detection because they think you're welcoming them
Replies (2)
-
@Brahvim@mastodon.gamedev.place 2026-08-10 02:11
@GroupNebula563@mastodon.social @Haste@mastodon.social Hmm. Perhaps a funny tag-spammed modern-day webapp that has no actually useful content, and plainer HTML pages with actual content? ...Or does it not work that way?
-
@marginalia@mastodon.social 2026-08-10 12:49
@GroupNebula563@mastodon.social @Haste@mastodon.social FWIW the idea is pretty good for getting rid of bots. I wouldn't try to block a bot or reject connections, that looks like something to be circumvented, an indicator for the scrapers to use a residental proxy and pretend to be chrome;' it's much better to serve a "Welcome to nginx!"-page or similar.