Post #1490130

2026-02-26 13:06 UTC

@pbloem That's a very good point. Thanks, that observation helps relieve some of my confusion. I hadn't considered that the transformer is dual trained as a text completer and an agent playing the "chat game" with its user, and that model can in fact embody both of these tasks in superposition. That's an interesting idea to hold. Still, this is a slippery notion. They're trained as chat agents. This training mostly involves nudging text completion responses to better align with social norms and what people want from an AI. That's not the same as training them as problem solving agents in, say, a game environment. So, I don't think it's reasonable to drop an LLM trained for chat into any RL setting and expect it to act as an RL agent. But I suppose you could fine tune an LLM in any RL setting, and basically what you get is a typical RL agent working with the latent space of a foundation model, which is... interesting.

Replies (2)

@ngaylinn@tech.lgbt 2026-02-26 13:11

@pbloem Extrapolating a bit, I think a good decision making AI needs to be trained and evaluated on a specific domain of decisions, using an appropriate model and training method. I think the promise of LLM agents is that they are "AGI" and can make decisions in any context by generalizing their experience. That's the part that seems most problematic to me. There's little reason to believe this besides wishful thinking.

Open ##1490131
@pbloem@sigmoid.social 2026-02-26 13:13

@ngaylinn Much of how they are trained is secret but we know that they've been trained on multiple specific tasks since the beginning. In fact this paper first showed that if you do that , they generalize to unseen tasks. This came before the idea of finetuning them to be chat bots. https://arxiv.org/abs/2109.01652 Since then, I think the AI labs have only increased the number of tasks. Things like coding and gaming are good because you can derive rewards automatically, so that's were the focus is.

Open ##1490132