Visualizing the decision boundaries and final result of the same problem optimized by Muon and by AdamW is fascinating. The solutions are qualitatively very different. AdamW has visibly redundant and useless neurons, Muon less so, but AdamW's reconstruction looks more generic.