Post #2357989

@jgordon@appdot.net @vrandecic@mas.to They don't. I've seen ppl having to close down their sites because their provider upped their plan by a lot of money bcs they were being crawled that much so that it caused so much traffic, it wasn't covered by private hosting plans anymore. robots.txt doesn't do anything anymore, those crawlers don't respect rules.