System and method for efficient evolution of deep convolutional neural networks using filter-wise recombination and propagated mutations
An efficient technique of machine learning is provided for training a plurality of convolutional neural networks (CNNs) with increased speed and accuracy using a genetic evolutionary model. A plurality of artificial chromosomes may be stored representing weights of artificial neuron connections of the plurality of respective CNNs. A plurality of pairs of the chromosomes may be recombined to generate, for each pair, a new chromosome (with a different set of weights than in either chromosome of the pair) by selecting entire filters as inseparable groups of a plurality of weights from each of the pair of chromosomes (e.g., “filter-by-filter” recombination). A plurality of weights of each of the new or original plurality of chromosomes may be mutated by propagating recursive error corrections incrementally throughout the CNN. A small random sampling of weights may optionally be further mutated to zero, random values, or a sum of current and random values.
1. A method of training a plurality of convolutional neural networks (CNNs) using a genetic evolutionary model, each CNN comprising a hierarchy of layers, each layer comprising one or more channels, each channel comprising a plurality of artificial neurons, each CNN further comprising a plurality of filters, each filter representing a group of a plurality of weights of connections between neurons in a pair of channels in layers of the hierarchy, the method comprising:
storing a plurality of artificial chromosomes representing the weights of the artificial neuron connections of the plurality of respective CNNs;
recombining a plurality of pairs of the chromosomes to generate, for each pair, a new chromosome representing a different set of weights than in either chromosome of the pair of artificial neuron connections of a new CNN;
mutating a plurality of weights in the different set of weights of each new chromosome by propagating error correction to correct errors in the different set of weights in the new CNN;
updating the plurality of artificial chromosomes by adding the new recombined and mutated chromosomes and removing at least some of the plurality of artificial chromosomes; and
iteratively repeating said updating until the updated plurality of artificial chromosomes represent weights of CNNs that have a convergent error.
2. The method of claim 1 , wherein mutating iteratively propagates error correction recursively layer-by-layer from a final output layer based on expected values of a training dataset backward until weights connecting a first layer to inputs are updated in the new CNN.
3. The method of claim 1 , wherein an initial weight w ij connecting neuron i to neuron j is mutated to a new weight w′ ij based on a function of the error of neuron j, ƒ(e j ), as w′ ij =w ij +ƒ(e j ).
4. The method of claim 3 , wherein the function of error is computed using a technique selected from the group consisting of: first-order error correction, second-order error correction, higher-than-second-order error correction, backpropagation, Hessian-Free Optimization, and contrastive-divergence.
5. The method of claim 1 comprising mutating a random sampling of the different set of weights of each new chromosome to zero, random values, or a sum of current and random values.
6. The method of claim 1 comprising mutating to zero all weights in each new chromosome representing an entire inseparable filter in a random sampling of filters of the new CNN.
7. The method of claim 1 , wherein mutating propagates error correction to only neurons in the CNN that are connected by weights that are randomly selected to be mutated.
8. The method of claim 1 , wherein mutating is executed in multiple error correction iterations.
9. The method of claim 1 , wherein mutating is executed using variable learning rate or momentum.
10. The method of claim 1 , wherein recombining comprises selecting entire filters as inseparable groups of a plurality of weights from each of the pair of chromosomes to recombine only weights of each entire selected filter together such that each selected filter cannot be divided during the recombining.
11. A system for training a plurality of convolutional neural networks (CNNs) using a genetic evolutionary model, each CNN comprising a hierarchy of layers, each layer comprising one or more channels, each channel comprising a plurality of artificial neurons, each CNN further comprising a plurality of filters, each filter representing a group of a plurality of weights of connections between neurons in a pair of channels in respective adjacent layers of the hierarchy, the system comprising:
one or more memories to store a plurality of artificial chromosomes representing the weights of the neuron connections of the plurality of respective CNNs; and
one or more processors configured to:
recombine a plurality of pairs of the chromosomes to generate, for each pair, a new chromosome representing a different set of weights than in either chromosome of the pair of artificial neuron connections of a new CNN,
mutate a plurality of weights in the different set of weights of each new chromosome by propagating error correction to correct errors in the different set of weights in the new CNN,
update the plurality of artificial chromosomes by adding the new recombined and mutated chromosomes and removing at least some of the plurality of artificial chromosomes, and
iteratively repeat said update until the updated plurality of artificial chromosomes represent weights of CNNs that have a convergent error.
12. The system of claim 11 , wherein the one or more processors are configured to iterate propagating error correction recursively layer-by-layer from a final output layer based on expected values of a training dataset backward until weights connecting a first layer to inputs are updated in the new CNN.
13. The system of claim 11 , wherein the one or more processors are configured to compute an initial weight w ij connecting neuron i to neuron j is mutated to a new weight w′ ij based on a function of the error of neuron j, ƒ(e j ), as w′ ij =w ij +ƒ(e j ).
14. The system of claim 13 , wherein the one or more processors are configured to compute the function of error using a technique selected from the group consisting of: first-order error correction, second-order error correction, higher-than-second-order error correction, backpropagation, Hessian-Free Optimization, and contrastive-divergence.
15. The system of claim 11 , wherein the one or more processors are configured to mutate a random sampling of the different set of weights of each new chromosome to zero, random values, or a sum of current and random values.
16. The system of claim 11 , wherein the one or more processors are configured to mutate to zero all weights in each new chromosome representing an entire inseparable filter in a random sampling of filters of the new CNN.
17. The system of claim 11 , wherein the one or more processors are configured to mutate by propagating error correction to only neurons in the CNN that are connected by weights that are randomly selected to be mutated.
18. The system of claim 11 , wherein the one or more processors are configured to mutate in multiple error correction iterations.
19. The system of claim 11 , wherein the one or more processors are configured to mutate using variable learning rate or momentum.
20. The system of claim 11 , wherein the one or more processors are configured to recombine by selecting entire filters as inseparable groups of a plurality of weights from each of the pair of chromosomes to recombine only weights of each entire selected filter together such that each selected filter cannot be divided during the recombination.