Post #484648

My #AI crawler maze has seen activity grow over the last 30d and is now handling in the region of 3 million requests a day The links into the maze are marked noindex,nofollow. The maze is also a disallow in robots.txt and each of the pages that the maze serves are marked noindex,nofollow. No honest bot should be in there. The number of crawlers executing javascript seems to have dropped off though - the various JS traps are firing much less frequently