System and method for efficient evolution of deep convolutional neural networks using filter-wise recombination and propagated mutations
An efficient technique of machine learning is provided for training a plurality of convolutional neural networks (CNNs) with increased speed and accuracy using a genetic evolutionary model. A plurality of artificial chromosomes may be stored representing weights of artificial neuron connections of the plurality of respective CNNs. A plurality of pairs of the chromosomes may be recombined to generate, for each pair, a new chromosome (with a different set of weights than in either chromosome of the pair) by selecting entire filters as inseparable groups of a plurality of weights from each of the pair of chromosomes (e.g., “filter-by-filter” recombination). A plurality of weights of each of the new or original plurality of chromosomes may be mutated by propagating recursive error corrections incrementally throughout the CNN. A small random sampling of weights may optionally be further mutated to zero, random values, or a sum of current and random values.
1. A method of training a plurality of convolutional neural networks (CNNs) using a genetic evolutionary model, each CNN comprising a hierarchy of layers, each layer comprising one or more channels, each channel comprising a plurality of artificial neurons, each CNN further comprising a plurality of filters, each filter representing a group of a plurality of weights of connections between neurons in a pair of channels in respective adjacent layers of the hierarchy, the method comprising:
storing a plurality of artificial chromosomes representing the weights of the artificial neuron connections of the plurality of respective CNNs;
recombining a plurality of pairs of the chromosomes to generate, for each pair, a new chromosome that includes a different set of weights than in either chromosome of the pair, wherein recombining comprises selecting entire filters as inseparable groups of a plurality of weights from each of the pair of chromosomes by traversing the length of the pair of chromosomes, progressing filter-by-filter, switching back and forth at random between selecting a filter from a first chromosome or a second chromosome in the pair;
updating the plurality of artificial chromosomes by adding the new chromosomes and removing at least some of the non-recombined chromosomes; and
iteratively repeating said updating until the updated plurality of artificial chromosomes represent weights of CNNs that have a convergent error.
2. The method of claim 1 comprising mutating a plurality of weights of the new chromosome based on errors computed for the connected neurons, the errors of the neurons computed recursively from the final output layer based on the expected values of a training dataset and propagating those error corrections backward incrementally to lower layers in the CNN.
3. The method of claim 1 comprising mutating a random plurality of individual weights or filters of multiple weights in the new chromosome to zero, to random values, or to a sum of the current value and a random value.
4. The method of claim 1 , wherein each filter is a one or two-dimensional filter representing the weights between a single pair of channels from adjacent layers.
5. The method of claim 1 , wherein recombining comprises selecting three-dimensional filters as inseparable groups of a plurality of weights between one channel of a first layer and a plurality of channels of an adjacent layer.
6. The method of claim 1 , wherein recombining comprises a hybrid of selecting entire filters and selecting individual weights in different training iterations, layers, channels, or segments of the new chromosome.
7. The method of claim 1 wherein the two chromosomes are selected according to a pre-defined measure of accuracy based on a training dataset.
8. The method of claim 7 , wherein the new chromosome generated in a first iteration is recombined in a subsequent iteration to generate a subsequent new chromosome.
9. A method of training a plurality of convolutional neural networks (CNNs) using a genetic evolutionary model, each CNN comprising a hierarchy of layers, each layer comprising one or more channels, each channel comprising a plurality of artificial neurons, each CNN further comprising a plurality of filters, each filter representing a group of a plurality of weights of connections between neurons in a pair of channels in respective adjacent layers of the hierarchy, the method comprising:
storing a plurality of artificial chromosomes representing the weights of the neuron connections of the plurality of respective CNNs;
mutating a plurality of weights of one of the plurality of chromosomes to form a new chromosome by updating the plurality of weights based on errors computed for the connected neurons, the errors of the neurons computed recursively from the final output layer based on the expected values of a training dataset and propagating those error corrections backward incrementally to lower layers in the CNN to only neurons in the CNN that are connected by weights that are randomly selected to be mutated;
updating the plurality of artificial chromosomes by adding the new chromosomes and removing at least some of the non-recombined chromosomes; and
iteratively repeating said updating until the updated plurality of artificial chromosomes represent weights of CNNs that have a convergent error.
10. The method of claim 9 , wherein an initial weight w ij connecting neuron i to neuron j is mutated to a new weight w′ ij based on a function of the error of neuron j, f(e j ) as w′ ij =w ij +f(e j ).
11. The method of claim 9 , wherein the weights are mutated by an operation selected from the group consisting of: backpropagation, gradient descent, Hessian-Free optimization, contrastive-divergence method, first-order method, second-order method, and third-order method.
12. A system for training a plurality of convolutional neural networks (CNNs) using a genetic evolutionary model, each CNN comprising a hierarchy of layers, each layer comprising one or more channels, each channel comprising a plurality of artificial neurons, each CNN further comprising a plurality of filters, each filter representing a group of a plurality of weights of connections between neurons in a pair of channels in respective adjacent layers of the hierarchy, the system comprising:
one or more memories to store a plurality of artificial chromosomes representing the weights of the neuron connections of the plurality of respective CNNs; and
one or more processors configured to:
recombine a plurality of pairs of the chromosomes to generate, for each pair, a new chromosome that includes a different set of weights than in either chromosome of the pair, wherein recombining comprises selecting entire filters as inseparable groups of a plurality of weights from each of the pair of chromosomes by traversing the length of the pair of chromosomes, progressing filter-by-filter, switching back and forth at random between selecting a filter from a first chromosome or a second chromosome in the pair,
update the plurality of artificial chromosomes by adding the new chromosomes and removing at least some of the non-recombined chromosomes, and
iteratively repeat said update until the updated plurality of artificial chromosomes represent weights of CNNs that have a convergent error.
13. The system of claim 12 , wherein the one or more processors are configured to mutate a plurality of weights of the new chromosome based on errors computed for the connected neurons, the errors of the neurons computed recursively from the final output layer based on the expected values of a training dataset and propagating those error corrections backward incrementally to lower layers in the CNN.
14. The system of claim 12 , wherein the one or more processors are configured to mutate a random plurality of individual weights or filters of multiple weights in the new chromosome to zero, to random values, or to a sum of the current value and a random value.
15. The system of claim 12 , wherein the one or more processors are configured to select each filter to be a one or two-dimensional filter representing the weights between a single pair of channels from adjacent layers.
16. The system of claim 12 , wherein the one or more processors are configured to recombine by selecting three-dimensional filters as inseparable groups of a plurality of weights between one channel of a first layer and a plurality of channels of an adjacent layer.
17. The system of claim 12 , wherein the one or more processors are configured to recombine using a hybrid of selecting entire filters and selecting individual weights in different training iterations, layers, channels, or segments of the new chromosome.
18. The system of claim 12 , wherein the one or more processors are configured to select the two chromosomes according to a pre-defined measure of accuracy based on a training dataset.
19. The system of claim 18 , wherein the one or more processors are configured to recombine the new chromosome generated in a first iteration to generate a subsequent new chromosome in a subsequent iteration.
20. A system for training a plurality of convolutional neural networks (CNNs) using a genetic evolutionary model, each CNN comprising a hierarchy of layers, each layer comprising one or more channels, each channel comprising a plurality of artificial neurons, each CNN further comprising a plurality of filters, each filter representing a group of a plurality of weights of connections between neurons in a pair of channels in respective adjacent layers of the hierarchy, the system comprising:
one or more memories to store a plurality of artificial chromosomes representing the weights of the neuron connections of the plurality of respective CNNs; and
one or more processors configured to:
mutate a plurality of weights of one of the plurality of chromosomes to form a new chromosome by updating the plurality of weights based on errors computed for the connected neurons, the errors of the neurons computed recursively from the final output layer based on the expected values of a training dataset and propagating those error corrections backward incrementally to lower layers in the CNN to only neurons in the CNN that are connected by weights that are randomly selected to be mutated,
update the plurality of artificial chromosomes by adding the new chromosomes and removing at least some of the non-recombined chromosomes, and
iteratively repeat said update until the updated plurality of artificial chromosomes represent weights of CNNs that have a convergent error.
21. The system of claim 20 , wherein the one or more processors are configured to mutate an initial weight w ij connecting neuron i to neuron j into a new weight w′ ij based on a function of the error of neuron j, f(e j ) as w′ ij =w ij +f(e j ).
22. The system of claim 20 , wherein the one or more processors are configured to mutate the weights by an operation selected from the group consisting of: backpropagation, gradient descent, Hessian-Free optimization, contrastive-divergence method, first-order method, second-order method, and third-order method.