@demofox@mastodon.gamedev.place
Post #1005863
2026-04-07 00:59 UTC
To me, the craziest part of gradient descent is that you take derivatives of a bunch of different parameters, some derivatives are bigger than others, and those sizes just happen to be the right proportion.
It seems like mixing multiple parameters to optimize should be some non linear thing.
Sure the gradient is only guaranteed linear for infinitessimal lengths, but still. Seems weird.
Replies (1)
-
@maxc@merveilles.town 2026-04-07 02:04
@demofox@mastodon.gamedev.place > It seems like mixing multiple parameters to optimize should be some non linear thing. I mean, they are, thats why you have to step it and re-evaluate and do so a million times, and why weights can blow up, haha But yeah it is kinda wild magic. Just attribute the change all the way back through the entire thing and then smoosh the numbers a little better