Post #1490133
2026-02-26 13:19 UTC
@pbloem This whole line of research makes me deeply uncomfortable.
I guess what I most like about RL is that they have a theoretical grounding for what decision making in a dynamic setting is, how to measure performance in that setting, and how to make algorithms that reliably converge on good performance.
Here, again, I would claim the agents do this task by interpolating decision making examples found in their training data. It's stunning that this works, to some extent, but it completely lacks that sort of rigorous grounding. We don't fully understand why or how LLMs pull off this trick, and we can only talk about their reliability in empirical terms.
Sometimes I'm okay with "it works, so who cares why?" But with decision making I really want assurances.
Replies (1)
-
@pbloem@sigmoid.social 2026-02-26 13:24
@ngaylinn This might be a disambiguation issue. What you're describing sounds like the old symbolic RL, where you get proofs and guarantees. What is called RL here is really just a gradient estimator, so you can backpropagate through a sampling process. There are no explicit states or modeling of the problem. Just an action space and a reward signal. I can certainly understand your discomfort. It's completely black box, and people are very rapidly putting it in charge of things.