Post #1490125
2026-02-20 15:17 UTC
@abucci That's a very interesting point!
I guess this is one drawback of how RL is founded in such simplified formulations. Most folks just assume "max returns" because that's how it started, that's how it's taught, and that's how it is almost always done in practice. But that only makes sense in a deterministic and finite context, which is rare in practice.
Building on what you're saying, there are surely scenarios where you're more interested in best-case or worst-case than average-case performance. Along with variance, this is something that's neglected.
Even variance doesn't necessarily get at the fullness of diversity. Assuming that there is a single scalar value to optimize is a massive simplification that mostly goes unnoticed.
Replies (1)
-
@abucci@buc.ci 2026-02-20 16:49
@ngaylinn@tech.lgbt See. doesn't that feel more productive already? As soon as you begin to poke at using average/sum total as an optimization goal, all sorts of interesting questions present themselves! And you're right to question the use of any single number, whether it's variance or something else. I'm nowhere near up enough on RL to know whether "multi-objective RL" or something like it is a thing (I'm sure people have written about this but has anyone really nailed it?). I do suspect there are lovely applications of coevolutionary algorithms to some of these questions, though (coevolve policies against subsets of states, for instance).