@acdha@code4lib.social and yes, the llm bot defenses are a real problem for web archives. I saw in IIPC Slack yesterday that they are considering this as a topic for a technical meeting at the next conference in a few weeks (meeting may be open to remote participation).
Cloudflare has a Verified Bot registry, which I think some web archiving orgs have gotten on?