A new paper is under construction, the results came in, and the abstract is ready.
Recurrent to Feedforward Transformation for Fast Decision-Making in Indexed Memories
Abstract: In this paper we analyse the performance of indexed memory systems for fast decision-making using a feed-foward interpretation of recurrent networks. We recapitulate the components of an indexed memory system, identifying high-impact neurons from input patterns and using localist plasticity to stabilize them as index neurons. We demonstrate the discoverability of natural categories by showing that white-noise simulations fail to develop natural indices. We also examine the relationship between pattern length and network size, showing that matching input to a fixed network size can be accomplished by pre-processing, which in biology corresponds to thalamo-cortical chunking.
We further characterize the emergence of deterministic classification (100% correctness). When analyzing pattern sets, we can delimit a core set from marginal patterns. For the core set, performance is deterministic and correct, even for previously unseen patterns, whereas marginal patterns are probabilistically classified. For MNIST and fashion MNIST patterns, only about 10% of patterns are marginal. We consider this core-margin structure a hallmark of natural cognition, fast, automated, and fully correct for core patterns, and probabilistic, error-prone only for marginal or borderline patterns.
To investigate fast decision-making in a trained network, we employ shortened latencies for index neurons and lateral inhibition and reduce feedback connections from index neurons. Biologically, this corresponds to neuromodulator-induced shifts in network properties, from a recurrent network to a mostly feedforward network. The resulting network performs decisions in \approx 40ms through feedforward selection of the correct index nodes. This enormous speed-up compared to the recurrent network used for training (\approx 400ms) is well in line with the requirements of biological organisms.
We conclude that a recurrent-to-feedforward shift in operation fits with the requirements of fast, errorless operation on the basis of a recurrently trained fixed size network, like a cortical module. Decision-making is very fast on the order of 40ms or less. Linking cortical modules will then be the route to more complex cognitive functions.
—
This is quite a nice result. It firmly places our work in the area of biological modeling, at the same time as providing a clear explanation for animal/human cognitive functioning beyond function approximation/machine learning/deep learning.