Parallelizing neural networks during training
A parallel convolutional neural network is provided. The CNN is implemented by a plurality of convolutional neural networks each on a respective processing node. Each CNN has a plurality of layers. A subset of the layers are interconnected between processing nodes such that activations are fed forward across nodes. The remaining subset is not so interconnected.
1. A system for parallelizing a neural network having a plurality of parameters during training of the neural network on a training set to determine a final parameter setting for the parameters of the neural network that produces correct classifications for the training set, the system comprising:
a plurality of parallel neural networks, wherein each of the plurality of parallel neural networks is implemented on a respective computing node, and wherein the plurality of parallel neural networks each receive a same input image from the training set and collectively generate an output that classifies the input image, wherein each of the neural networks comprises a respective plurality of layers, wherein each plurality of layers comprises an interconnected layer and a non-interconnected layer, and wherein processing data through the layers of each of the plurality of parallel neural networks comprises:
providing output from the interconnected layer to at least one layer in each of the other parallel neural networks in the plurality of parallel neural networks; and
providing output from the non-interconnected layer only to a layer of the same parallel neural network, and wherein the system is configured to:
after the training, store the final parameter setting for the parameters of the neural network on one or more non-transitory computer storage media.
2. The system of claim 1 , wherein processing data through the layers of each of the plurality of parallel neural networks further comprises: providing output from the interconnected layer to at least one layer of the same parallel neural network.
3. The system of claim 1 , wherein each of the plurality of layers comprises a respective plurality of nodes and wherein each node generates a respective output activation based on an input activation received from one or more other layers.
4. The system of claim 3 , wherein providing output from the interconnected layer to at least one layer of at least one different parallel neural network of the plurality of parallel neural networks comprises providing output activations from each node of the interconnected layer to each node in at least one layer of a subset of the other parallel neural networks in the plurality of parallel neural networks.
5. The system of claim 3 , wherein providing output from the interconnected layer to at least one layer of at least one different parallel neural network of the plurality of parallel neural networks comprises providing output activations from only a subset of the nodes of the interconnected layer to a subset of the nodes in at least one layer of at least one different parallel neural network in the plurality of parallel neural networks.
6. The system of claim 1 , wherein the parallel neural networks are convolutional neural networks.
7. The system of claim 1 , wherein the non-interconnected layer is a convolutional layer.
8. The system of claim 1 , wherein the interconnected layer is a fully-connected layer.
9. A method for parallelizing a neural network having a plurality of parameters during training of the neural network on a training set to determine a final parameter setting for the parameters of the neural network that produces correct classifications for the training set, the method comprising:
processing data using each of a plurality of parallel neural networks, wherein each of the plurality of parallel neural networks is implemented on a respective computing node, wherein the plurality of parallel neural networks each receive a same input image from the training set and collectively generate an output that classifies the input image, wherein each of the neural networks comprises a respective plurality of layers, wherein each plurality of layers comprises an interconnected layer and a non-interconnected layer, wherein processing data using each of the plurality of parallel neural networks comprises processing the data through the layers of each of the plurality of parallel neural networks, and wherein processing the data through the layers of each of the plurality of parallel neural networks comprises:
providing output from the interconnected layer to at least one layer in each of the other parallel neural networks in the plurality of parallel neural networks;
providing output from the non-interconnected layer only to a layer of the same parallel neural network; and
after the training, storing the final parameter setting for the parameters of the neural network on one or more non-transitory computer storage media.
10. The method of claim 9 , wherein processing data through the layers of each of the plurality of parallel neural networks further comprises: providing output from the interconnected layer to at least one layer of the same parallel neural network.
11. The method of claim 9 , wherein each of the plurality of layers comprises a respective plurality of nodes and wherein each node generates a respective output activation based on an input activation received from one or more other layers.
12. The method of claim 11 , wherein providing output from the interconnected layer to at least one layer of at least one different parallel neural network of the plurality of parallel neural networks comprises providing output activations from each node of the interconnected layer to each node in at least one layer of a subset of the other parallel neural networks in the plurality of parallel neural networks.
13. The method of claim 11 , wherein providing output from the interconnected layer to at least one layer of at least one different parallel neural network of the plurality of parallel neural networks comprises providing output activations from only a subset of the nodes of the interconnected layer to a subset of the nodes in at least one layer of at least one different parallel neural network in the plurality of parallel neural networks.
14. The method of claim 9 , wherein the parallel neural networks are convolutional neural networks.
15. The method of claim 9 , wherein the non-interconnected layer is a convolutional layer.
16. The method of claim 9 , wherein the interconnected layer is a fully-connected layer.
17. A non-transitory computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations for parallelizing a neural network having a plurality of parameters during training of the neural network on a training set to determine a final parameter setting for the parameters of the neural network that produces correct classifications for the training set, the operations comprising:
processing data using each of a plurality of parallel neural networks, wherein each of the plurality of parallel neural networks is implemented on a respective computing node, wherein the plurality of parallel neural networks each receive a same input image from the training set and collectively generate an output that classifies the input image, wherein each of the neural networks comprises a respective plurality of layers, wherein each plurality of layers comprises an interconnected layer and a non-interconnected layer, wherein processing data using each of the plurality of parallel neural networks comprises processing the data through the layers of each of the plurality of parallel neural networks, and wherein processing the data through the layers of each of the plurality of parallel neural networks comprises:
providing output from the interconnected layer to at least one layer in each of the other parallel neural networks in the plurality of parallel neural networks;
providing output from the non-interconnected layer only to a layer of the same parallel neural network; and
after the training, storing the final parameter setting for the parameters of the neural network on one or more non-transitory computer storage media.
18. The non-transitory computer storage medium of claim 17 , wherein processing data through the layers of each of the plurality of parallel neural networks further comprises: providing output from the interconnected layer to at least one layer of the same parallel neural network.
19. The non-transitory computer storage medium of claim 17 , wherein each of the plurality of layers comprises a respective plurality of nodes and wherein each node generates a respective output activation based on an input activation received from one or more other layers.
20. The non-transitory computer storage medium of claim 19 , wherein providing output from the interconnected layer to at least one layer of at least one different parallel neural network of the plurality of parallel neural networks comprises providing output activations from each node of the interconnected layer to each node in at least one layer of a subset of the other parallel neural networks in the plurality of parallel neural networks.
21. The non-transitory computer storage medium of claim 19 , wherein providing output from the interconnected layer to at least one layer of at least one different parallel neural network of the plurality of parallel neural networks comprises providing output activations from only a subset of the nodes of the interconnected layer to a subset of the nodes in at least one layer of at least one different parallel neural network in the plurality of parallel neural networks.
22. The non-transitory computer storage medium of claim 17 , wherein the parallel neural networks are convolutional neural networks.
23. The non-transitory computer storage medium of claim 17 , wherein the non-interconnected layer is a convolutional layer.
24. The non-transitory computer storage medium of claim 17 , wherein the interconnected layer is a fully-connected layer.