2026-09-20 05:10 UTC
Reddit blocked my scraper with a 403, so I built a collector for 6 open APIs instead (Python)
I was building a small tool to collect posts and comments about a topic and save them as JSON. The plan was to use Reddit, but scripts get a 403 from the .json endpoint that works fine in the browser. I tried the legitimate things (own User-Agent, full headers) and even a client that mimics Chrome. All blocked, and the official API needs prior approval. I decided not to work around it.
So I switched to sources with open APIs: Hacker News, Lemmy, Stack Exchange, Dev.to, Mastodon and Discourse. Each one is a small Python module that returns posts in the same format, so the output is one JSON file organized as source > language > posts > comments. There is also a FastAPI service and a language filter (langdetect).
A few things I learned:
“0 results” isn’t always an error: Dev.to and Mastodon only search by tag.
Stack Exchange allows 300 requests per day without a key, so I fetch answers in batches.
Text-search APIs require all words: on Hacker News, “most popular frontend language” matches 11 stories, “frontend language” matches 793.
Do you know other open APIs that are good for tech discussions, especially non-English ones? I’d like to test them.
Replies (0)
No replies.