Feedback connections in neural networks for layer reuse

Abstract

The increasing interest and integration of artificial intelligence (AI) have resulted in expanded artificial neural network (ANN) sizes following the proposed scaling laws [1-3] regarding their performance. Consequently, to accommodate practical deployment and widespread accessibility, the deep learning (DL) field and therein ANNs could benefit from revised architectures. Drawing inspiration from the biological brain and recurrent neural networks (RNNs), we propose feedback connections in ANNs, leveraging existing weights by looping subsequent outputs back to earlier layers. In this manner, networks can further iterate on their latent space representations of inputs without introducing additional trainable parameters. Here, we express novel approaches to implement said connections in ANNs with two different ways of backpropagation through these. Through the custom-built neurons framework [4], we easily evaluate and compare variations of feedback connections. While the framework has certain constraints regarding computational efficiency for large networks or datasets, we successfully demonstrate the benefits of feedback connections for specific problem settings. Furthermore, we probe these trained networks, thus assessing the performance impact and information flow of feedback loops. As we show promising results on small datasets, we expect this to scale accordingly, following the success of similar studies [5,6]. Consequently, as networks implementing feedback connections mimic deeper networks while retaining a relatively small parameter footprint with comparable or even improved performance, we suspect upscaling our novel approaches to implement this may make the future of AI more accessible and deployable.