Evolving recurrent networks using genetic programming

1. A computer-implemented system for evolving a recurrent neural network (RNN) that solves a provided problem, the system comprising:

a memory storing a candidate RNN genome database having a pool of candidate RNN nodes, each of the candidate RNN nodes identifying respective values for a plurality of hyperparameters of the candidate RNN node and each of the candidate RNN nodes representing a neural network as a tree structure, each candidate RNN node having a unique tree structure with respect to the other candidate RNN nodes of the pool of candidate RNN nodes;

an assembly module that assembles N RNN layers by:

selecting a candidate RNN node from the pool of candidate RNN nodes, the candidate RNN node being selected such that each respective RNN layer includes the selected candidate RNN node, N being a predetermined integer that is greater than 1, and

replicating the selected candidate RNN node within each respective RNN layer M−1 times, such that each respective RNN layer includes M candidate RNN nodes, M being a predetermined integer that is greater than 1;

an evolution module that evolves, for each respective RNN layer, the M candidate RNN nodes G times, G being a predetermined integer that is greater than 1, the M candidate RNN nodes of each respective RNN layer being evolved by:

performing speciation and crossover at a probability of sc and according to a speciation compatibility threshold of cth,

performing an insert mutation at a probability of im,

performing a shrink mutation at a probability of sm, and

performing a replacement mutation at a probability of rm;

a training module that trains the evolved candidate RNN nodes of each of the N RNN layers using training data;

an evaluation module that evaluates a performance of each trained candidate RNN node of each RNN layer using validation data and assigns a fitness value to each candidate RNN node;

a competition module that forms an elitist pool of candidate RNN nodes in dependence on their assigned fitness values; and

a solution harvesting module providing for deployment of RNN layers instantiated with candidate RNN nodes from the elitist pool.

2. A computer-implemented system for evolving a recurrent neural network (RNN) that solves a provided problem, the system comprising:

a memory storing a candidate RNN genome database having a pool of candidate RNN nodes, each of the candidate RNN nodes identifying respective values for a plurality of hyperparameters of the candidate RNN node and each of the candidate RNN nodes representing a neural network as a tree structure, each candidate RNN node having a unique tree structure with respect to the other candidate RNN nodes of the pool of candidate RNN nodes;

an assembly module that assembles N RNN layers by:

selecting a candidate RNN node from the pool of candidate RNN nodes, the candidate RNN nodes being selected such that each respective RNN layer includes the selected candidate RNN node, N being a predetermined integer that is greater than 1, and

replicating the selected candidate RNN node within each respective RNN layer M−1 times, such that each respective RNN layer includes M candidate RNN nodes, M being a predetermined integer that is greater than 1;

an evolution module that evolves, for each respective RNN layer, the M candidate RNN nodes G times, G being a predetermined integer that is greater than 1, the M candidate RNN nodes of each respective RNN layer being evolved by:

performing speciation and crossover at a probability of sc and according to a speciation compatibility threshold of cth,

performing an insert mutation at a probability of im,

performing a shrink mutation at a probability of sm, and

performing a replacement mutation at a probability of rm;

a training module that trains the evolved candidate RNN nodes of each of the N RNN layers using training data;

an evaluation module that:

evaluates a performance of each trained candidate RNN node of each layer using validation data for X epochs to obtain perplexity of each candidate RNN node at X epochs, X being an integer that is greater than 1,

implements one or more long short-term memory (LSTM) neural network trained to predict perplexity of each candidate RNN node at Y epochs, given the obtained perplexity at X epochs as an input, Y being an integer and at least 4 times greater than X, and

assigns a fitness value to each candidate RNN node in dependence upon the predicted perplexity;

a competition module that forms an elitist pool of candidate RNN nodes in dependence on their assigned fitness values; and

a solution harvesting module providing for deployment of RNN layers instantiated with candidate RNN nodes from the elitist pool.

3. A computer-implemented system for evolving a recurrent neural network (RNN) that solves a provided problem, the system comprising:

a memory storing a candidate RNN genome database having a pool of candidate RNN nodes, each of the candidate RNN nodes identifying respective values for a plurality of hyperparameters of the candidate RNN node and each of the candidate RNN nodes representing a neural network as a tree structure, each candidate RNN node having a unique tree structure with respect to the other candidate RNN nodes of the pool of candidate RNN nodes;

an assembly module that assembles N RNN layers by:

selecting, for each respective RNN layer of the N RNN layers, a predetermined number H of candidate RNN nodes from the pool of candidate RNN nodes, the H candidate RNN nodes being selected such that each of the H candidate RNN nodes has a unique structure, N being a predetermined integer that is greater than 1, and

replicating the H candidate RNN nodes within each respective RNN layer a certain number of times, such that each RNN layer includes the same number of candidate RNN nodes;

an evolution module that evolves, for each respective RNN layer, the H candidate RNN nodes G times, G being a predetermined integer that is greater than 1, the H candidate RNN nodes of each respective RNN layer being evolved by:

performing speciation and crossover at a probability of sc and according to a speciation compatibility threshold of cth,

performing an insert mutation at a probability of im,

performing a shrink mutation at a probability of sm, and

performing a replacement mutation at a probability of rm;

a training module that trains the evolved candidate RNN nodes of each of the N RNN layers using training data;

an evaluation module that evaluates a performance of each trained candidate RNN node of each RNN layer using validation data and assigns a fitness value to each candidate RNN node;

a competition module that forms an elitist pool of candidate RNN nodes in dependence on their assigned fitness values; and

a solution harvesting module providing for deployment of RNN layers instantiated with candidate RNN nodes from the elitist pool.

4. A computer-implemented system for evolving a recurrent neural network (RNN) that solves a provided problem, the system comprising:

a memory storing a candidate RNN genome database having a pool of candidate RNN nodes, each of the candidate RNN nodes identifying respective values for a plurality of hyperparameters of the candidate RNN node and each of the candidate RNN nodes representing a neural network as a tree structure, each candidate RNN node having a unique tree structure with respect to the other candidate RNN nodes of the pool of candidate RNN nodes;

an assembly module that assembles N RNN layers by:

selecting, for each respective RNN layer of the N RNN layers, a predetermined number of H candidate RNN node from the pool of candidate RNN nodes, the H candidate RNN nodes being selected such that each of the H candidate RNN nodes has a unique structure, N being a predetermined integer that is greater than 1, and

replicating the H candidate RNN node within each respective RNN layer a certain number of times, such that each RNN layer includes the same number of candidate RNN nodes;

an evolution module that evolves, for each respective RNN layer, the H candidate RNN nodes G times, G being a predetermined integer that is greater than 1, the H candidate RNN nodes of each respective RNN layer being evolved by:

performing speciation and crossover at a probability of sc and according to a speciation compatibility threshold of cth,

performing an insert mutation at a probability of im,

performing a shrink mutation at a probability of sm, and

performing a replacement mutation at a probability of rm;

a training module that trains the evolved candidate RNN nodes of each of the N RNN layers using training data;

an evaluation module that:

evaluates a performance of each trained candidate RNN node of each RNN layer using validation data for X epochs to obtain perplexity of each candidate RNN node at X epochs, X being an integer that is greater than 1,

implements one or more long short-term memory (LSTM) neural network trained to predict perplexity of each candidate RNN node at Y epochs, given the obtained perplexity at X epochs as an input, Y being an integer and at least 4 times greater than X, and

assigns a fitness value to each candidate RNN node in dependence upon the predicted perplexity;

a competition module that forms an elitist pool of candidate RNN nodes in dependence on their assigned fitness values; and

a solution harvesting module providing for deployment of RNN layers instantiated with candidate RNN nodes from the elitist pool.