Post #3247496

@axolotl@kolektiva.social that said, to answer your question more directly: hard to tell. right now, the bot has found 20k links and crawled 5k, but as it crawls links, it finds more, so that is far from 25% done... we are careful about crawling substack because they have rate limiting, so this is a slow process. i'd say "dozens of hours" or "days". once the site is crawled, it also takes some amount of time to trickle back into the internet archive and, from there, the wayback machine