Post #1708063

Hi #infosec community: any recommended practices for preventing a personal blog webpage from being dominated by crawlers? The robots.txt seems largely ignored. Crawlers presently consume ~75% of resources according to the access logs. I don't mind not being listed on any web search engines. Would a basic auth work? Can one make the basic auth dialog show a message that provides the password hint in it, like "What is my first name?" #blogging