System and method for addressing overfitting in a neural network
A system for training a neural network. A switch is linked to feature detectors in at least some of the layers of the neural network. For each training case, the switch randomly selectively disables each of the feature detectors in accordance with a preconfigured probability. The weights from each training case are then normalized for applying the neural network to test data.
1. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for training a neural network with reduced overfitting and in reduced time, the neural network having a plurality of layers, each of the layers including one or more feature detectors each having a corresponding set of weights, and the operations comprising:
obtaining, by one or more computers, a plurality of training cases; and
training, by the one or more computers, the neural network on the plurality of training cases with reduced overfitting and reduced co-adaptation of feature detectors by reducing reliance of feature detectors in the neural network on other feature detectors in the neural network, comprising:
for a first training case of the plurality of training cases:
determining a first set of one or more feature detectors to disable during processing of the first training case,
disabling the first set of one or more feature detectors in accordance with the determining,
processing the first training case using the neural network with the first set of one or more feature detectors disabled to generate a predicted output for the first training case, thereby reducing overfitting and reducing co-adaptation of feature detectors by reducing reliance on the first set of one or more feature detectors by other feature detectors in the neural network; and
after processing the first training case using the neural network with the first set of one or more feature detectors disabled and for a second training case of the plurality of training cases:
determining a second, different set of one or more feature detectors to disable during processing of the second training case,
disabling the second, different set of one or more feature detectors in accordance with the determining,
processing the second training case with at least one of the feature detectors in the first set of feature detectors enabled and the second, different set of feature detectors disabled to generate a predicted output for the second training case, thereby reducing overfitting and co-adaptation of feature detectors by reducing reliance on the second set of one or more feature detectors by other feature detectors in the neural network.
2. The system of claim 1 , wherein a subset of the feature detectors are associated with respective probabilities of being disabled during processing of each of the training cases, and wherein determining the first set of one or more feature detectors to disable during processing of the first training case comprises:
determining whether to disable each of the feature detectors in the subset based on the respective probability associated with the feature detector.
3. The system of claim 2 , wherein training the neural network further comprises:
adjusting the weights of each of the feature detectors in the neural network to generate trained values for each weight in the set of weights corresponding to the feature detector.
4. The system of claim 3 , the operations further comprising:
normalizing the trained weights for each of the feature detectors in the subset, wherein normalizing the trained weights comprises multiplying the trained values of the weights for each of the one or more feature detectors in the subset by a respective probability of the feature detector not being disabled during processing of each of the training cases.
5. The system of claim 2 , wherein the subset includes feature detectors in a first layer of the plurality of layers and feature detectors in one or more second layers of the plurality of layers, wherein the feature detectors in the first layer are associated with a first probability and wherein the feature detectors in the one or more second layers are associated with a second, different probability.
6. The system of claim 5 , wherein the first layer is an input layer of the neural network and the one or more second layers are hidden layers of the neural network.
7. The system of claim 5 , wherein the first layer and the one or more second layers are hidden layers of the neural network.
8. The system of claim 1 , wherein determining the first set of one or more feature detectors to disable during processing of the first training case comprises:
determining to disable the same feature detectors that were disabled during processing of a preceding training case.
9. The system of claim 1 , wherein training the neural network comprises training the neural network to perform speech recognition.
10. The system of claim 1 , wherein training the neural network comprises training the neural network to perform image classification.
11. The system of claim 1 , wherein training the neural network comprises training the neural network to perform object recognition.
12. A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that when executed by one or more computers cause the one or more computers to perform operations for reducing overfitting during training of a neural network, the operations comprising:
obtaining, by one or more computers, a plurality of training cases; and
training, by the one or more computers, the neural network on the plurality of training cases with reduced overfitting and reduced co-adaptation of feature detectors by reducing reliance of feature detectors in the neural network on other feature detectors in the neural network, comprising:
for a first training case of the plurality of training cases:
determining a first set of one or more feature detectors to disable during processing of the first training case,
disabling the first set of one or more feature detectors in accordance with the determining,
processing the first training case using the neural network with the first set of one or more feature detectors disabled to generate a predicted output for the first training case, thereby reducing overfitting and co-adaptation of feature detectors by reducing reliance on the first set of one or more feature detectors by other feature detectors in the neural network; and
after processing the first training case using the neural network with the first set of one or more feature detectors disabled and for a second training case of the plurality of training cases:
determining a second, different set of one or more feature detectors to disable during processing of the second training case,
disabling the second, different set of one or more feature detectors in accordance with the determining,
processing the second training case with at least one of the feature detectors in the first set of feature detectors enabled and the second, different set of feature detectors disabled to generate a predicted output for the second training case, thereby reducing overfitting and co-adaptation of feature detectors by reducing reliance on the second set of one or more feature detectors by other feature detectors in the neural network.
13. The non-transitory computer storage medium of claim 12 , wherein training the neural network comprises training the neural network to perform speech recognition.
14. The non-transitory computer storage medium of claim 12 , wherein training the neural network comprises training the neural network to perform image classification.
15. The non-transitory computer storage medium of claim 12 , wherein training the neural network comprises training the neural network to perform object recognition.
16. The non-transitory computer storage medium of claim 12 , wherein a subset of the feature detectors are associated with respective probabilities of being disabled during processing of each of the training cases, and wherein determining the first set of one or more feature detectors to disable during processing of the first training case comprises:
determining whether to disable each of the feature detectors in the subset based on the respective probability associated with the feature detector.
17. The non-transitory computer storage medium of claim 16 , wherein training the neural network further comprises:
adjusting the weights of each of the feature detectors in the neural network to generate trained values for each weight in the set of weights corresponding to the feature detector.
18. The non-transitory computer storage medium of claim 17 , the operations further comprising:
normalizing the trained weights for each of the feature detectors in the subset, wherein normalizing the trained weights comprises multiplying the trained values of the weights for each of the one or more feature detectors in the subset by a respective probability of the feature detector not being disabled during processing of each of the training cases.
19. The non-transitory computer storage medium of claim 16 , wherein the subset includes feature detectors in a first layer of the plurality of layers and feature detectors in one or more second layers of the plurality of layers, wherein the feature detectors in the first layer are associated with a first probability and wherein the feature detectors in the one or more second layers are associated with a second, different probability.