Post #2300718

2026-05-09 19:52 UTC

Thank you! Let me wildly oversimplify and make sure I understand. The fundamental problem is that if you train on a set that includes multiple independent facts, the generative aspect of the model - the ability to generate new text that is statistically consistent with the training data - requires remixing and combining tokens in a way that will inevitably result in factual errors. Like, if your training data includes “all men are mortal” and “all lions are cats” then in order to generate new text it has to be “loose” enough to output “all men are cats”. Feedback and reinforcement can adjust the probabilities to a degree, but because the model is fundamentally about token probabilities and doesn’t have any other way of accounting for whether a statement is actually true, there’s no way to completely remove it. You can reinforce that “all cats are mortal” is a better answer, but you can’t train it that “all men are cats” is invalid.

Replies (1)

@scruiser@awful.systems 2026-05-09 21:41

You’ve described the problem with generalization yes. Well, you could maybe sort of train it not to generate “all men are cats”, but then that might also prevent it from making the more correct generalization “all cats are mortal” or even completely valid generalizations like combing “all men are mortal” and “Socrates is man” to get “Socrates is mortal”. The problem with monofacts is a bit more subtle. Let’s say the fact that “John Smith was born in Seattle in 1982, earned his PhD from Stanford in 2008, and now leads AI research at Tech Corp,” appears only once in the training data set. Some of the other words the model will have seen multiple times and be able to generate tokens in the right way for. Like Seattle as a location in the US, Stanford as a college, 2008 as a date, etc. But the combination describing a fact about John Smith appearing uniquely trains the model to try to generate facts that are unique combinations of data. So the model might try to make up a fact like “Jane Doe was born in Omaha in 1984, earned her master from Caltech in 2006, and is now CEO of Tech Corp” because it fits the pattern of a unique fact that was in its training data set.

Open ##2305003