Post #366772

2026-02-20 14:11 UTC

Reinforcement Learning is AI for playing games, planning actions, and doing things in the real world. It's extremely powerful, but also a fun and interesting topic to learn. I'm teaching myself now. I was always intimidated by all the esoteric jargon and math in RL. The classic Sutton and Barto text clears that up nicely, though. It's one of the best text books I have ever studied, and it reveals RL is much less complex than it sounds. I think the excessive mathiness comes from the question "is this policy optimal?" which is a natural place to start, and has been fruitful for the field. Except, in practice it's rare we can find optimal policies for practical problems. Also, our mathematical formalisms often don't fit, which is why there are so many variations based on what information is available, and what you want to focus on. This produces the tangled mess of names and notations, but really it's just the same idea from subtly different angles. #rl #ai #programming

Replies (1)

@abucci@buc.ci 2026-02-20 14:46

@ngaylinn@tech.lgbt One of my favorite sets of RL-related research questions has to do with what exactly is meant by "optimal" policy. One of the simplest and most common answers is that the "optimal" policy returns the highest reward on average (equivalently in total; with perhaps some fiddling around with what the average or sum is taken over and whether it's weighted). Averages are dubious metrics to me, so I always question them when I see them. A more interesting question to ask is what's the policy with the highest average reward that also has the minimum variance? And then: (when) does it make sense to trade average reward against variance, sacrificing a bit of reward in order to achieve a more robust policy? I saw a neat talk at ICML 2013 (I think?) where the person asked exactly this, and showed that on a maze-solving task, the low-variance policies had much more "natural" looking paths through the maze than the high-reward ones. This is clearly dependent on a lot of technical details I'm omitting, and I don't know that it'd be a generalizeable observation. Still, it was great they asked this question and made some observations about it.

Open ##1490124