Post #1151294

LLMs generate the next most probable token given the previous context of tokens they have (not an average of the entire internet). And post-training shifts the odds a bit further in a relatively useful direction. So given the right context the LLM will mostly consistently regurgitate content stolen from PhDs and academic papers, maybe even managing to shuffle it around in a novel way that is marginally useful. Of course, that is only the general trend given the right^tm^ prompt. Even with a prompt that looks mostly right, one seemingly innocuous word in the wrong place might nudge the odds and you get the answer of a moron /r/hypotheticalphysics in response to a physics question. Or a asking for a recipe gets you elmer’s glue on your mozarella pizza from a reddit joke answer. So tldr; you’re right, but since it is possible to get somewhat better than average internet junk with pre-training and prompting, llm boosters and labs have convinced themselves they are just a few more iterations of training approaches and prompting techniques away from entirely eliminating the problem, when the best they can do is make it less likely.