Post #1490128
2026-02-26 13:00 UTC
Replies (3)
-
@pbloem@sigmoid.social 2026-02-26 13:01
@ngaylinn This is not new either. All of them going back to ChatGPT have been trained by some form of RL in real-world tasks.
-
@ngaylinn@tech.lgbt 2026-02-26 13:06
@pbloem That's a very good point. Thanks, that observation helps relieve some of my confusion. I hadn't considered that the transformer is dual trained as a text completer and an agent playing the "chat game" with its user, and that model can in fact embody both of these tasks in superposition. That's an interesting idea to hold. Still, this is a slippery notion. They're trained as chat agents. This training mostly involves nudging text completion responses to better align with social norms and what people want from an AI. That's not the same as training them as problem solving agents in, say, a game environment. So, I don't think it's reasonable to drop an LLM trained for chat into any RL setting and expect it to act as an RL agent. But I suppose you could fine tune an LLM in any RL setting, and basically what you get is a typical RL agent working with the latent space of a foundation model, which is... interesting.
-
@abucci@buc.ci 2026-02-26 13:55
@pbloem@sigmoid.social @ngaylinn@tech.lgbt What does the word "agent" mean in this context? By my read folks are replacing one inscrutable buzzword with another as though that changes anything important. Calling the guts of a so-called intelligent agent a next-high-reward-action predictor instead of a next-token predictor doesn't change the paucity of the model nor its fundamental and currently-unavoidable limitations. The "percepts" are biased, filtered, and digitized; the possible "actions" are as well; and the guts perform a seq2seq transformation. Yes it's digitized and filtered: FP4, FP8, FP16 etc. are not real numbers. That's the fundamental issue, not what labels we slap on the components or how complicated the wiring between them is. This is the reason LLMs and anything like RLHF that wraps them are ill-suited to most decisionmaking tasks along multiple critical axes whenever the decisions touch the real world. Joseph Weizenbaum was already writing about this 50 years ago. Decisions that matter are not "games" in any sense of that word. See also: small vs. large words (Leonard Savage) and the notion of Knightian uncertainty; the Chinese Room thought experiment (John Searle); the Mary's Room thought experiment (Frank Jackson); the octopus test (Bender and Koller: https://aclanthology.org/2020.acl-main.463.pdf ); and the notions of undermining and overmining (Graham Harman) for a start. I'm not a fan of his "snake oil" framing, but Arvind Narayanan is onto something when he says "this product is essentially an elaborate random number generator" in 2019: https://www.cs.princeton.edu/~arvindn/talks/MIT-STS-AI-snakeoil.pdf (video of a similar talk from 2021, which is worth watching: https://www.cs.princeton.edu/news/how-recognize-ai-snake-oil ).