Crazy inventor. on arch btw, sometimes a crab. Will build stuff for coffee. Find me pounding servers in the basement. If my instance at kampftoast.de should be down you can find me here.
Crazy inventor. on arch btw, sometimes a crab.
Will build stuff for coffee.
Find me pounding servers in the basement.
If my instance at kampftoast.de should be down you can find me here.
Posts
Crazy inventor. on arch btw, sometimes a crab. Will build stuff for coffee. Find me pounding servers in the basement. If my instance at kampftoast.de should be down you can find me here.
Crazy inventor. on arch btw, sometimes a crab. Will build stuff for coffee. Find me pounding servers in the basement. If my instance at kampftoast.de should be down you can find me here.
Thinking of building a semantic search engine based on vector embedding (SBERT, probably self trained models in the future based on what was indexed to disconnect from the bigtech stuff) and parallel full text search. What language to use?
It will be separated into 3 parts:
- Crawler/Data importer (Importing public data dumps, well behaved RSS and HTTP crawler)
- Parser (Splitting paragraphs/sentences, calculating embedding vectors, keyword extraction),
- Frontend (Web interface/API, full-text-search, similarity search).
Parts could be different languages. If you have an opinion why they could be please reply :)
DB will be Postgres with pg_vectors extension for vector indexing and ts_vector full text search.
The idea would be to let the user decide if they want a similarity search or (exact) full text search, as well as providing an "similar to this" option for search results (which is missing from current search engine implementations IMHO)
Basically I want the "drill down"-effect to allow for old-school web browsing/discovery that was provided by web directories in the past but have it automatically discover more pages.
:reply_request: :boost_request:
Crazy inventor. on arch btw, sometimes a crab. Will build stuff for coffee. Find me pounding servers in the basement. If my instance at kampftoast.de should be down you can find me here.
Crazy inventor. on arch btw, sometimes a crab. Will build stuff for coffee. Find me pounding servers in the basement. If my instance at kampftoast.de should be down you can find me here.
Crazy inventor. on arch btw, sometimes a crab. Will build stuff for coffee. Find me pounding servers in the basement. If my instance at kampftoast.de should be down you can find me here.
Crazy inventor. on arch btw, sometimes a crab. Will build stuff for coffee. Find me pounding servers in the basement. If my instance at kampftoast.de should be down you can find me here.
Crazy inventor. on arch btw, sometimes a crab. Will build stuff for coffee. Find me pounding servers in the basement. If my instance at kampftoast.de should be down you can find me here.
Crazy inventor. on arch btw, sometimes a crab. Will build stuff for coffee. Find me pounding servers in the basement. If my instance at kampftoast.de should be down you can find me here.
Crazy inventor. on arch btw, sometimes a crab. Will build stuff for coffee. Find me pounding servers in the basement. If my instance at kampftoast.de should be down you can find me here.
Crazy inventor. on arch btw, sometimes a crab. Will build stuff for coffee. Find me pounding servers in the basement. If my instance at kampftoast.de should be down you can find me here.
Crazy inventor. on arch btw, sometimes a crab. Will build stuff for coffee. Find me pounding servers in the basement. If my instance at kampftoast.de should be down you can find me here.
Crazy inventor. on arch btw, sometimes a crab. Will build stuff for coffee. Find me pounding servers in the basement. If my instance at kampftoast.de should be down you can find me here.