System and method for parallelizing convolutional neural networks
A parallel convolutional neural network is provided. The CNN is implemented by a plurality of convolutional neural networks each on a respective processing node. Each CNN has a plurality of layers. A subset of the layers are interconnected between processing nodes such that activations are fed forward across nodes. The remaining subset is not so interconnected.
1. One or more non-transitory computer readable media storing instructions that when executed by one or more computers cause the one or more computers to implement a convolutional neural network, the convolutional neural network comprising:
a sequence of neural network layers, wherein the sequence of neural network layers comprises:
a first convolutional layer configured to receive a first convolutional layer input derived from an input image and to process the first convolutional layer input to generate a first convolved output;
a first max-pooling layer immediately after the first convolutional layer in the sequence configured to pool the first convolved output to generate a first pooled output;
a second convolutional layer immediately after the max-pooling layer in the sequence configured to receive the first pooled output and to process the first pooled output to generate a second convolved output,
a plurality of fully-connected layers after the second convolutional layer in the sequence configured to receive an output derived from the second convolved output and to collectively process the output derived from the second convolved output to generate an initial output for the input image, and
an output layer configured to classify the input image based at least in part on the initial output.
2. The computer readable media of claim 1 , wherein the plurality of fully-connected layers are not immediately after the second convolutional layer in the sequence.
3. The computer readable media of claim 1 , wherein the second convolutional layer is not immediately followed by a max-pooling layer in the sequence.
4. The computer readable media of claim 1 , wherein the sequence of neural network layers comprises one or more other layers between the second convolutional layers and the plurality of fully-connected layers.
5. The computer readable media of claim 1 , wherein the sequence of neural network layers comprises one or more non max-pooling layers immediately after the second convolutional layer in the sequence.
6. The computer readable media of claim 1 , further comprising:
one or more other sequences of neural network layers, each other sequence comprising layers identical to the first convolutional layer, the first max-pooling layer, the second convolutional layer, and the plurality of fully-connected layers.
7. The computer readable media of claim 6 , wherein at least one of the plurality of fully-connected layers is configured to provide an output to a layer in the at least one other sequence of neural network layers.
8. The computer readable media of claim 6 , wherein the second convolutional layer is configured to provide the second convolved output only to another layer in the sequence of neural network layers and not to any layers in any of the other sequences of neural network layers.
9. The computer readable media of claim 6 , wherein each other sequence of neural network layers is configured to collectively process the input image to generate a respective initial output for the input image.
10. The computer readable media of claim 9 , wherein the output layer is configured to receive the initial outputs for each sequence of layers and to classify the input image by processing the sequence outputs.
11. One or more non-transitory computer readable media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for processing an input image using a convolutional neural network comprising a sequence of neural network layers, the operations comprising:
processing a first convolutional layer input derived from the input image using a first convolutional layer in the sequence, wherein the first convolutional layer is configured to receive the first convolutional layer input and to process the first convolutional layer input to generate a first convolved output;
processing the first convolved output using a first max-pooling layer immediately after the first convolutional layer in the sequence, wherein the first max-pooling layer is configured to pool the first convolved output to generate a first pooled output;
processing the first pooled output using a second convolutional layer immediately after the max-pooling layer in the sequence, wherein the second convolutional layer is configured to receive the first pooled output and to process the first pooled output to generate a second convolved output;
processing an output derived from the second convolved output using a plurality of fully-connected layers after the second convolutional layer in the sequence, wherein the plurality of fully-connected layers are configured to receive the output derived from the second convolved output and to collectively process the output derived from the second convolved output to generate an initial output for the input image; and
processing the initial output using an output layer, wherein the output layer is configured to classify the input image based at least in part on the initial output.
12. The computer readable media of claim 11 , wherein the plurality of fully-connected layers are not immediately after the second convolutional layer in the sequence.
13. The computer readable media of claim 11 , wherein the second convolutional layer is not immediately followed by a max-pooling layer in the sequence.
14. The computer readable media of claim 11 , wherein the sequence of neural network layers comprises one or more other layers between the second convolutional layers and the plurality of fully-connected layers.
15. The computer readable media of claim 11 , wherein the sequence of neural network layers comprises one or more non max-pooling layers immediately after the second convolutional layer in the sequence.
16. The computer readable media of claim 11 , wherein the neural network system further comprises one or more other sequences of neural network layers, each other sequence comprising layers identical to the first convolutional layer, the first max-pooling layer, the second convolutional layer, and the plurality of fully-connected layers.
17. The computer readable media of claim 16 , wherein at least one of the plurality of fully-connected layers is configured to provide an output to a layer in the at least one other sequence of neural network layers.
18. The computer readable media of claim 6 , wherein the second convolutional layer is configured to provide the second convolved output only to another layer in the sequence of neural network layers and not to any layers in any of the other sequences of neural network layers.
19. The computer readable media of claim 16 , the operations further comprising processing the input image using each other sequence of neural network layers, wherein each other sequence of neural network layers is configured to collectively process the input image to generate a respective initial output for the input image.
20. The computer readable media of claim 19 , wherein the output layer is configured to receive the initial outputs for each sequence of layers and to classify the input image by processing the sequence outputs.