Post #2723511
2026-05-12 08:24 UTC
@felwert@fedihum.org Not sure if it's best practice or just old-school: honest user agent (name of the software, version number, possibly the harvesting purpose or person responsible), check robots.txt for delay/interval instructions before harvesting, use generous delays/intervals between requests if no instructions are given, and never fire async requests.
This is how parts of the data for the NFDI4Culture Knowledge Graph are harvested 🤖
Replies (1)
-
@felwert@fedihum.org 2026-05-12 08:53
@jonatan@mastodon.social Thanks! Yes, that’s the line in which I was thinking as well, so good to see it confirmed.