Post #1367465

A while ago, I've set-up the ai.robots.txt nginx config to return a 403 to all AI crawlers based on their user-agent. I thought it'd be ineffective as these crawlers are known to switch user-agents when being blocked, but it worked fine and significantly dropped the amount of requests I was getting (once the crawlers had emptied their queues). Now that I'm switching to something else than nginx, I haven't yet taken the time to look at solutions similar to the one I just described, but as I migrated my git server from my old infra to my new one, I quickly checked the logs out of curiosity... And crawlers were already hammering my server, opening blames and downloading my code left and right... It was only back open for less than a minute! ​:dumpster_fire:​