Multi-objective evolutionary neural architecture search for recurrent neural networks

Loading...
Thumbnail Image

Authors

Booysen, Reinhard
Bosman, Anna

Journal Title

Journal ISSN

Volume Title

Publisher

Springer

Abstract

Artificial neural network (NN) architecture design is a nontrivial and time-consuming task that often requires a high level of human expertise. Neural architecture search (NAS) serves to automate the design of NN architectures and has proven to be successful in automatically finding NN architectures that outperform those manually designed by human experts. NN architecture performance can be quantified based onmultiple objectives,which include model accuracy and some NN architecture complexity objectives, among others. The majority of modern NAS methods that consider multiple objectives for NN architecture performance evaluation are concerned with automated feed forward NN architecture design, which leaves multi-objective automated recurrent neural network (RNN) architecture design unexplored. RNNs are important for modeling sequential datasets, and prominent within the natural language processing domain. It is often the case in real world implementations of machine learning and NNs that a reasonable trade-off is accepted for marginally reduced model accuracy in favour of lower computational resources demanded by the model. This paper proposes a multi-objective evolutionary algorithm-based RNN architecture search method. The proposed method relies on approximate network morphisms for RNN architecture complexity optimisation during evolution. The results show that the proposed method is capable of finding novel RNN architectures with comparable performance to state-of-the-art manually designed RNN architectures, but with reduced computational demand.

Description

DATA AVAILABILITY: 1. The Penn Treebank dataset used for the word-level NLP task in Sect. 4.1 is available for download at: https://github.com/reinn-cs/rnn-nas/tree/master/example_datasets/ptb/data. 2. The dataset used for the sequence learning task in Sect. 4.2 is artificially generated as described in the relevant section. The source code for the generation of the dataset is included in the source code repository of the MOE/RNAS algorithm implementation, which is available at: https://github.com/reinn-cs/rnn-nas. 3. The data used for the analysis of the MOE/RNAS algorithm was based on the experimental results obtained after implementing the MOE/RNAS algorithm to search for and optimise RNN architectures for the respective datasets. The source code for the MOE/RNAS algorithm implementation is available at: https://github.com/reinn-cs/rnn-nas.

Keywords

Evolutionary algorithms, Artificial neural network (ANN), Neural architecture search (NAS), Recurrent neural network (RNN), SDG-09: Industry, innovation and infrastructure

Sustainable Development Goals

SDG-09: Industry, innovation and infrastructure

Citation

Booysen, R. & Bosman, A.S. 2024, 'Multi-objective evolutionary neural architecture search for recurrent neural networks', Neural Processing Letters, vol. 56, art. 200, pp. 1-31. https://DOI.org/10.1007/s11063-024-11659-0.