Post #1398076
2026-04-08 04:35 UTC
@danderson @ocean I've looked through the scraper (last change 7 years ago), and fixed things up rough-and-dirty (will clean up and push it to GitHub there too, shortly). The scraper runs, but since ArchiveTeam has no jobs out, it just idles. I guess it will need central coordination, or maybe I've missed something? 🤔
Replies (2)
-
@danderson@hachyderm.io 2026-04-08 04:41
@imrehg @ocean yeah I think once a working scraper build is up, there also needs to be something generating seed URLs to start from (iirc the wiki suggested starting with the sitemap?), and then archiveteam people need to push some buttons to open up the new job and maybe move some of the swarm over to it? Not sure, I've never MC'd a crawl just observed other AT crawls happen
-
@jmjl@fedi.tilde.green 2026-05-03 18:42
@imrehg@fosstodon.org @danderson@hachyderm.io @ocean@raru.re Please don't run modified code on the production system, only run approved-by-AT code. Also, we tried to get what we could via https://wiki.archiveteam.org/index.php/Distributed_recursive_crawls (apparently not documented on the wiki page as of posting this)