Post #3195796

@neuralreckoning@neuromatch.social @GabrielBena@neuromatch.social Thanks for sharing, solid work. Some thoughts on this point: "we see that resource constrained biological brains are great at generalisation, an expected outcome of having specialised modules with generalisable functions, while machine learning systems are not. Maybe we give them too much computational power?" There seem to be \approx zero generalization penalties from overparameterization in networks trained via SGD. Eg: train a 4-layer MLP on MNIST. Performance is identical if the two hidden layers are anywhere from 128 - 16k wide. Is it true that resource constraints engender better generalization in biological networks? (Trained, presumably, without SGD?). I see no plots or metrics of generalization - ? Regarding functional specialization, imho how networks learn is a interplay between 'natural' spectral / eigenmode / Fourier-coefficient learning (axis-agnostic), and more interpretable axis-aligned learning, as driven by asymmetries in e.g. Adam or regularizers (or biology). Again, in practice forcing functional specialization tends to *negatively* impact ANN generalization performance; instead, it seems to be primarily a function the networks structure, and not of the quantity of gross computation therein (above a limit). The former which, of course, you've varied, in an interesting way... so where's the generalization figure? :-)