Post #1490124

2026-02-20 14:46 UTC

@ngaylinn@tech.lgbt One of my favorite sets of RL-related research questions has to do with what exactly is meant by "optimal" policy. One of the simplest and most common answers is that the "optimal" policy returns the highest reward on average (equivalently in total; with perhaps some fiddling around with what the average or sum is taken over and whether it's weighted). Averages are dubious metrics to me, so I always question them when I see them. A more interesting question to ask is what's the policy with the highest average reward that also has the minimum variance? And then: (when) does it make sense to trade average reward against variance, sacrificing a bit of reward in order to achieve a more robust policy? I saw a neat talk at ICML 2013 (I think?) where the person asked exactly this, and showed that on a maze-solving task, the low-variance policies had much more "natural" looking paths through the maze than the high-reward ones. This is clearly dependent on a lot of technical details I'm omitting, and I don't know that it'd be a generalizeable observation. Still, it was great they asked this question and made some observations about it.

Replies (1)

@ngaylinn@tech.lgbt 2026-02-20 15:17

@abucci That's a very interesting point! I guess this is one drawback of how RL is founded in such simplified formulations. Most folks just assume "max returns" because that's how it started, that's how it's taught, and that's how it is almost always done in practice. But that only makes sense in a deterministic and finite context, which is rare in practice. Building on what you're saying, there are surely scenarios where you're more interested in best-case or worst-case than average-case performance. Along with variance, this is something that's neglected. Even variance doesn't necessarily get at the fullness of diversity. Assuming that there is a single scalar value to optimize is a massive simplification that mostly goes unnoticed.

Open ##1490125