Post #216512

2026-02-02 08:05 UTC

@screwlisp@gamerplus.org I remember doing something like this as a grad student when I first started learning about natural language processing. I was just playing around with various ideas in code (I was using Perl at the time). I remember breaking-up a sentence into tokens and just counting how often one token followed another in a large corpus of data. Then I created a simple weighted directed graph data structure associating every token with every other token weighted by the count, that is, how often any one token X was seen immediately after any other token Y. The directed graph was really a dictionary data structure where the token X was the key and the value associated with the key was another dictionary associating the token Y with it’s weight, so there was no matrix, or I guess you could say it was a sparse matrix. I even wrote an “inference” algorithm where you would just perform a random walk through the graph biased by the weights. The output was gibberish, but (to my surprise) mostly grammatically correct. I wanted to scale it up, associating tokens triples with other token triples to see if that improved the grammar, but I lost interest. This is a little different from how LLMs works though, in one important way: an LLM also encodes the position of the tokens in the prompt and uses that position information as something to be trained on. You could do this with the symbolic process I described above (the “sparse matrix), but you would have to take an input list of tokens X and an output list of tokens Y, and run the “training” process for each element in the list, so X[0] -> Y[0] would have it’s own sparse matrix, X[1] -> Y[1] would have it’s own sparse matrix, and so on. You would create a sparse matrix for each element in your list for some number of tokens (e.g. a thousand). Then it would really be like an LLM. Or did I just describe exactly what you have done here in Common Lisp? @AmenZwa@mathstodon.xyz @dougmerritt@mathstodon.xyz @kentpitman@climatejustice.social @rwxrwxrwx@mathstodon.xyz @aartaka@merveilles.town @neauoire@merveilles.town

Replies (1)

@ramin_hal9001@fe.disroot.org 2026-02-02 08:39

@screwlisp@gamerplus.org I should also say, LLMs also work on non-textual information like images and and sound. That is the advantage of using purely statistical methods, rather than the symbolic method you described. You can easily interface a LLM with a neural network that decomposes an image or sound into quanta that can be mapped onto the vectors that encode some “meaning” of words/tokens in a sentence, so you can prompt an LLM to create an image from text, or to create a textual description of an image. Statistical methods can also create a “sentiment” vector for passages of text, finding some way to encode the meaning of a textual passage into a number that you can use to compare to other numbers to decide if two passages are similar in meaning. This is useful for quickly searching through a large text corpus. Most modern chat bots do this, it is what they call “Retrieval Augmented Generation.” They use sentiment analysis to find text passages that seem like they may be related to your prompt, and use the “similar in sentiment” text passages to augment your prompt before then constructing an output. The purely symbolic language models with sparse matrices would probably be much more energy efficient, but the trade-off is that you lose the ability interface with neural networks trained on image or audio signals and also the ability to perform sentiment analysis. @AmenZwa@mathstodon.xyz @aartaka@merveilles.town @dougmerritt@mathstodon.xyz @kentpitman@climatejustice.social @neauoire@merveilles.town @rwxrwxrwx@mathstodon.xyz

Open ##216986