Post #2471349

2026-05-12 08:01 UTC

Fedi, I need your input! Since the web is dead and googling &quot;which user agent to use for responsible web scraping&quot; mostly returns AI-generated garbage promoting how to spoof the user agent for non-responsible web scraping: What are your best practices? Any guide you would recommend? #FediHelp #WebScraping #DigitalHumanities

Replies (4)

@jonatan@mastodon.social 2026-05-12 08:24

@felwert@fedihum.org Not sure if it's best practice or just old-school: honest user agent (name of the software, version number, possibly the harvesting purpose or person responsible), check robots.txt for delay/interval instructions before harvesting, use generous delays/intervals between requests if no instructions are given, and never fire async requests. This is how parts of the data for the NFDI4Culture Knowledge Graph are harvested 🤖

Open ##2723511
@apsmith@fediscience.org 2026-05-12 10:05

@felwert@fedihum.org The user agent string is arbitrary. If you are running your own software, give it an identifying name that makes sense and include a contact email address if possible.

Open ##2723513
@lobid@openbiblio.social 2026-05-12 10:30

@felwert@fedihum.org Wir schreiben in unseren "Richtlinien einer rücksichtsvollen lobid-Nutzung" was für einen User Agent wir uns wünschen: https://lobid.org/usage-policy/#user-agent-setzen

Open ##2723514
@bradr@infosec.exchange 2026-05-12 11:07

@felwert@fedihum.org I don't know about norms, but I find it helpful if the bot includes a url to a web page describing its purpose.

Open ##2723515