Post #3030814
2025-05-16 14:21 UTC
@tao@mathstodon.xyz do you see it as a possible step towards a tool (or generally speaking, "algorithm", ) that can eventually replace optimizers such as gradient descent or adam in large neural network training?
Replies (1)
-
@tao@mathstodon.xyz 2025-05-16 15:08
@kailuowang@mathstodon.xyz This is certainly plausible, especially for large-scale tasks in which one does not have enough expert human supervision available to manually adjust hyperparameters for each of the individual component subtasks. Or this sort of tool might be deployed as a "meta-optimization" layer on top of these existing tools, in which they decide how to select what combination of these tools to use, and what choices of hyperparameters to give those tools.