2026-05-27 05:02 UTC
nieman labs, a journalism watchdog, put up a detailed and well-meaning article about the tension between increasingly corporate journalism and the internet archive's drive to snapshot the entire web. they scanned hundreds of news sites to see who was using robots.txt to block the internet archive, and lots of them were.
but the archive doesn't care about your robots.txt file. they ignore them outright, as they 'do not necessarily serve our archival purposes'. this has been their policy for nearly a decade. so while these files do show intent to block, they're not actually blocking the internet archive at all.
Replies (0)
No replies.