Post #3247494
2026-03-12 17:04 UTC
Replies (3)
-
@Anarcat@kolektiva.social 2026-03-12 17:09
@axolotl@kolektiva.social well the site is already at least *partly* crawled by the internet archive, see https://web.archive.org/web/2/https://anarchosolarpunk.substack.com/ in fact, it looks like it already has decent coverage; my archive bot run is merely a safety measure to make sure we have *exhaustive* coverage
-
@Anarcat@kolektiva.social 2026-03-12 17:11
@axolotl@kolektiva.social that said, to answer your question more directly: hard to tell. right now, the bot has found 20k links and crawled 5k, but as it crawls links, it finds more, so that is far from 25% done... we are careful about crawling substack because they have rate limiting, so this is a slow process. i'd say "dozens of hours" or "days". once the site is crawled, it also takes some amount of time to trickle back into the internet archive and, from there, the wayback machine
-
@Anarcat@kolektiva.social 2026-03-12 17:11
@axolotl@kolektiva.social so if you don't want to touch substack but read this, check out the wayback machine *now* and let me know what's missing, maybe?