@neuralreckoning@neuromatch.social
Post #1442136
2025-01-14 13:06 UTC
@marcusghosh We compared the performance of linear and nonlinear algorithms that ignore temporal structure to two architectures that can use it. The first just uses a sliding window or fixed length short term memory. The second is a recurrent neural network, which in principle can have a much longer memory.
Replies (1)
-
@neuralreckoning@neuromatch.social 2025-01-14 13:07
@marcusghosh We were expecting the RNN to hugely outperform the sliding window approach, as it has a potentially longer memory and orders of magnitude more trainable parameters. However, if the bursts of information are not too long, the much simpler network does better. They also differ in how they generalise. If you train on one burst length and test on other burst lengths, the sliding window algorithms generalise well to longer bursts than they were trained on, and poorly to shorter bursts. The RNNs simply generalise worse the bigger the difference.