Post #3620259
2026-07-06 01:27 UTC
@ireneista@adhd.irenes.space i have done a small amount of research into how this could be done. you can definitely go with https://commoncrawl.org/ to start out with, instead of crawling sites yourself.
i feel like the governance and funding is the bigger issue, though. search engines become shit when they become ad engines. if i were to operate something like this, it would either have to be a money sink hobby thing i give only to friends, or an explicit "pay a subscription for access" type thing.
Replies (1)
-
@TommyTorty10@infosec.exchange 2026-07-06 05:13
@astrid@woem.space neato! I've been cobbling together a little toy search engine that will be based on my personal browsing history and bookmarks, cuz I didn't know things like this existed. I was going with history and bookmarks, cuz crawling is harder since the AI nonsense, even when you're doing it politely, and my rural internet is sloooww.