Post #3147648
2026-04-07 00:47 UTC
Replies (3)
-
@norikawa@efdn.club 2026-04-07 00:49
Also the people who made the frequency table type things (properly the term is word embedding) apparently remarked at how unexpected and unexplained it was that using an algorithm that explicitly associates words with each other based on how often they show up near one another produces vaguely reasonable predictions of what the possible next words are
-
@norikawa@efdn.club 2026-04-07 00:58
To be more detailed: Words are converted into tokens, which are just numbers representing either a single word or part of a word. So like "posted" would get split into a token for "post" and a token for "ed" Then these are converted into word embeddings, and then that conversion is somewhat reversed by running it back through the embedding. This outputs a provability distribution basically saying "hey for that given set of nearby words, here's the likely starting words for that", which can possibly tie synonyms to each other instead of having them act entirely separate I think Then that gets pumped through a whole bunch of encoder-decoder layers, which are partially just "here's a normal neural network layer" and partially attention layers, which are somehow trained (no one ever mentions the technique used here hmmMMMM) to output a weighted set of numbers basically representing the relative "importance" of words in the prompt like for "the red apple" the word "apple" is probably most important I guess? It's far and away the most vague and handwavey part of the whole setup and seems to just boil down to "we use a bunch of neural network layers to overcomplicate things enough til the monkeys on typewriters give us good output" anyway then you stack a bunch of encoder-decoder layers on top of each other, which by the way the encoder part here is just a separate stack of layers that gets its output pumped into the input of every decoder layer to try to mitigate problems I think??? "It just works" is very much the running theme when it comes to asking why anyone is doing it this way instead of another way so get used to not having answers Anyway after we've fed our word embedding distribution salad of nonsense through enough monkeys on typewriters, we end up with our final product: a list of tokens and their probabilities This is then selected from completely at random (with a bias towards high probability results) to output the next token Then we rerun the entire thing again but with the new token as part of the input and keep doing this for each and every token until we get a specially designated stopper token as output, or we output the preset maximum number of tokens for a single input
-
@cam@efdn.club 2026-04-07 00:50
@norikawa@efdn.club I honestly have never been impressed with the ML philosophy of “just bullshit a bunch of neural network layers until it works”