Post #3848947

@hal_pomeranz@infosec.exchange sorry, I was at dinner when you obvs tried to summon me. OpenSearch, a fork of ElasticSearch, is built atop Lucene, Doug Cutting's venerable classical inverted index library. An inverted index works much like an index in the back of a book, and like an index in the back of a book, it requires tokenization of text into "words". The text is split into tokens, the tokens are encoded as numbers, and they're turned into an inverted index, where each number taking the place of a word then has a list of other numbers corresponding to the documents containing that word (called a postings list). From there everyone has fun with different integer compression schemes. The standard text for this kind of search engine is Manning, available free online: https://nlp.stanford.edu/IR-book/information-retrieval-book.html. Mike McCandless added the finite-state transducer support to Lucene a few (2013) years ago: https://blog.mikemccandless.com/2010/12/using-finite-state-transducers-in.html. Next time we meet up I'll tell you about search engines that do support untokenized regex, and the ghastly costs they incur as a result.