Post #1398077
2026-04-08 04:41 UTC
@imrehg @ocean yeah I think once a working scraper build is up, there also needs to be something generating seed URLs to start from (iirc the wiki suggested starting with the sitemap?), and then archiveteam people need to push some buttons to open up the new job and maybe move some of the swarm over to it? Not sure, I've never MC'd a crawl just observed other AT crawls happen
Replies (1)
-
@imrehg@fosstodon.org 2026-04-09 01:32
@danderson @ocean I think whatever was it's gone. Following the suggestions in the wiki to get a sitemap https://wiki.archiveteam.org/index.php/Angelfire#Discovery_&_Downloading and that robots.txt is getting a 502 server error, that's probably a shutdown..