Post #2725916

@aaron@chirp.zadzmo.org bravo, I was considering creating something like this myself after I heard about some AI crawlers ignoring robots.txt. I'd combine this with a robots.txt that keeps away well behaved search engine crawlers and only punishes the bad actors.