Post #1490145

@pbloem@sigmoid.social @ngaylinn@tech.lgbt There are a number of misrepresentations in this comparison. I feel like going through some of them, since there is so much misinformation about LLMs and "agents" floating around and a bunch of that thrives on imprecision. Firstly, what is being described as an LLM is not an LLM--a Large LANGUAGE Model--because the output distribution is over a set of dates, not over next tokens (which are usually words or fragments of words). If you "ask" (prompt) a trained LLM for someone's birthday, it'll likely output a sequence of tokens that might include a date. Whether or not that date corresponds to the birthday of the person mentioned in the prompt is always a matter of chance (aka a hallucination) because the LLM has no knowledge representation, only symbol sequence correlations. Correlation is not causation, as the saying goes, but knowledge goes beyond mere correlation. It's been argued that correlation-only information is an expression of total ignorance! You could certainly train a statistical model to output a distribution over dates, and you could represent the model with a neural network based architecture. You could even call that an LLM if you like. But if it's a deep neural network trained past the interpolation threshold, as these things tend to be these days, then it almost surely would not output a uniform distribution. In this case that would mean all output unit activations would be roughly equal, which is dubious. It'd take an extraordinarily special set of weights, or a special architecture, to induce a neural network to output uniform distributions for all inputs it hasn't been trained with (it'd basically have to "blur" or "fuzz" all these inputs). More than that, neural networks are used for these sorts of tasks exactly because they can interpolate between inputs they have not been trained on. If "Ted's birthday is 01.01.1982" is in the training data, and you input a query representing "what is Ned's birthday", you'd expect higher-than-average weight on the 01.01.1982 output(s). Saying this won't happen is assuming the consequence! Reinforcement learning will not output "I don't know" unless the policy space explicitly represents an "I don't know" option as one of the available actions. Whether or not the RL-trained policy ever outputs this action is a function of what it encounters during training, what the reward function looks like, etc etc etc--in short, it depends on a lot of details the programmer explicitly builds into the model and training procedure. Claiming that RL somehow does this on its own is missing the point: if you rig the system to do this, then it might; if you don't, then it probably won't. In short, I don't think the claim that reinforcement learning somehow requires a shift in the mental model of LLMs or "agents" holds up to scrutiny.