Post #1490132

2026-02-26 13:13 UTC

@ngaylinn Much of how they are trained is secret but we know that they've been trained on multiple specific tasks since the beginning. In fact this paper first showed that if you do that , they generalize to unseen tasks. This came before the idea of finetuning them to be chat bots. https://arxiv.org/abs/2109.01652 Since then, I think the AI labs have only increased the number of tasks. Things like coding and gaming are good because you can derive rewards automatically, so that's were the focus is.

Replies (1)

@ngaylinn@tech.lgbt 2026-02-26 13:19

@pbloem This whole line of research makes me deeply uncomfortable. I guess what I most like about RL is that they have a theoretical grounding for what decision making in a dynamic setting is, how to measure performance in that setting, and how to make algorithms that reliably converge on good performance. Here, again, I would claim the agents do this task by interpolating decision making examples found in their training data. It's stunning that this works, to some extent, but it completely lacks that sort of rigorous grounding. We don't fully understand why or how LLMs pull off this trick, and we can only talk about their reliability in empirical terms. Sometimes I'm okay with "it works, so who cares why?" But with decision making I really want assurances.

Open ##1490133