Superpixel methods for convolutional neural networks
Methods, systems, and apparatus for efficiently performing a computation of a convolutional neural network layer. One of the methods includes transforming a X by Y by Z input tensor into a X′ by Y′ by Z′ input tensor, wherein X′ is smaller than or equal to X, Y′ is smaller than or equal to Y, and Z′ is larger than or equal to Z; obtaining one or more modified weight matrices, wherein the modified weight matrices operate on the X′ by Y′ by Z′ input tensor to generate a U′ by V′ by W′ output tensor, and the U′ by V′ by W′ output tensor comprises a transformed U by V by W output tensor; and processing the X′ by Y′ by Z′ input tensor using the modified weight matrices to generate the U′ by V′ by W′ output tensor.
1. A computer implemented method for training a convolutional neural network that includes a plurality of convolutional neural network layers on training data,
wherein the convolutional neural network comprises (i) at least one superpixel convolutional neural network layer having respective superpixel convolutional neural network layer parameters that is configured to receive an X by Y by Z input tensor and process the received input tensor using one or more convolutional neural network layer weight matrices to generate an U by V by W output tensor, and (ii) one or more other convolutional neural network layers having respective layer parameters,
wherein the method comprises:
obtaining training data; and
training the convolutional neural network on the training data to adjust the values of the parameters of the superpixel convolutional neural network and the values of the parameters of the other convolutional neural network layers to trained values, and
wherein receiving, by the superpixel convolutional neural network layer, an X by Y by Z input tensor and processing the received input tensor using one or more convolutional neural network layer weight matrices to generate an U by V by W output tensor comprises:
transforming the X by Y by Z input tensor into a X′ by Y′ by Z′ input tensor, wherein X′ is smaller than or equal to X, Y′ is smaller than or equal to Y, and Z′ is larger than or equal to Z;
obtaining one or more modified convolutional neural network layer weight matrices, wherein (i) the modified weight matrices operate on the X′ by Y′ by Z′ input tensor to generate a U′ by V′ by W′ output tensor, and (ii) the U′ by V′ by W′ output tensor comprises a transformed U by V by W output tensor, wherein U′ is smaller than or equal to U, V′ is smaller than or equal to V, and W′ is larger than or equal to W; and
processing the X′ by Y′ by Z′ input tensor using the modified weight matrices to generate the U′ by V′ by W′ output tensor, wherein the U′ by V′ by W′ output tensor comprises the U by V by W output tensor.
2. The method of claim 1 , wherein X′ is equal to a ceiling of X divided by a natural number N, Y′ is equal to a ceiling of Y divided by a natural number M, and Z′ is equal to Z multiplied by NM, wherein the ceiling corresponds to a smallest integer larger or equal to.
3. The method of claim 2 , wherein the one or more modified weight matrices comprise NM copies of each original weight corresponding to the entries of the modified weight matrices.
4. The method of claim 3 , wherein training the convolutional neural network on the training data comprises performing a gradient descent and backpropagation neural network training process.
5. The method of claim 4 , wherein performing the backpropagation training process comprises adding weight gradients across all NM copies to determine a net change in value for each original weight.
6. The method of claim 2 , wherein the X by Y by Z input tensor represents XY first inputs and the X′ by Y′ by Z′ input tensor represents X′Y′ super inputs with each super input comprising a plurality of first inputs.
7. The method of claim 6 , wherein the number of the plurality of first inputs equals NM inputs.
8. The method of claim 2 , wherein the operation of the modified weight matrices on the X′ by Y′ by Z′ input tensor is equivalent to the operation of the superpixel convolutional neural network layer weight matrices on the X by Y by Z input tensor.
9. The method of claim 2 , wherein entries of the modified weight matrices comprise one or more convolutional neural network layer weight matrices.
10. The method of claim 2 , wherein the modified weight matrices comprise matrices with a banded structure.
11. The method of claim 2 , wherein U′ is equal to a ceiling of U divided by a natural number P, V′ is equal to a ceiling of V divided by a natural number Q, and W′ is equal to W multiplied by PQ.
12. The method of claim 11 , wherein the convolutional neural network layer includes a stride S in the X dimension and stride T in the Y dimension, and wherein the relationship between stride S, T, output tensor U by V by W and transformed output tensor U′ by V′ by W′ is given by U′=ceiling(U/NS) and V′=ceiling(V/MT).
13. The method of claim 12 , wherein the U by V by W output tensor represents UV first outputs and the U′ by V′ by W′ output tensor represents U′V′ super outputs with each super output comprising a plurality of first outputs.
14. The method of claim 13 , wherein the plurality of first outputs equals PQ outputs.
15. The method of claim 1 , wherein the superpixel convolutional neural network layer comprises a pooling sub layer.
16. The method of claim 2 , wherein the U by V by W output tensor represents UV first outputs and the U′ by V′ by W′ output tensor represents U′ V′ super outputs with each super output comprising a plurality of first outputs, and
wherein the number of first outputs in the plurality of first outputs is dependent on the dimensions of the superpixel convolutional neural network layer weight matrices.
17. The method of claim 2 , wherein the U by V by W output tensor represents UV first outputs and the U′ by V′ by W′ output tensor represents U′ V′ super outputs with each super output comprising a plurality of first outputs, and
wherein the number of outputs in the plurality of first outputs is dependent on one or more of (i) the architecture of the superpixel convolutional neural network layer, (ii) an architecture of the convolutional neural network including the superpixel convolutional neural network layer, or (iii) a device implementing the superpixel convolutional neural network layer.
18. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations for training a convolutional neural network that includes a plurality of convolutional neural network layers on training data,
wherein the convolutional neural network comprises (i) at least one superpixel convolutional neural network layer having respective superpixel convolutional neural network layer parameters that is configured to receive an X by Y by Z input tensor and process the received input tensor using one or more convolutional neural network layer weight matrices to generate an U by V by W output tensor, and (ii) one or more other convolutional neural network layers having respective layer parameters, the operations comprising:
obtaining training data; and
training the convolutional neural network on the training data to adjust the values of the parameters of the superpixel convolutional neural network and the values of the parameters of the other convolutional neural network layers to trained values, and
wherein receiving, by the superpixel convolutional neural network layer, an X by Y by Z input tensor and processing the received input tensor using one or more convolutional neural network layer weight matrices to generate an U by V by W output tensor comprises:
transforming the X by Y by Z input tensor into a X′ by Y′ by Z′ input tensor, wherein X′ is smaller than or equal to X, Y′ is smaller than or equal to Y, and Z′ is larger than or equal to Z;
obtaining one or more modified convolutional neural network layer weight matrices, wherein (i) the modified weight matrices operate on the X′ by Y′ by Z′ input tensor to generate a U′ by V′ by W′ output tensor, and (ii) the U′ by V′ by W′ output tensor comprises a transformed U by V by W output tensor, wherein U′ is smaller than or equal to U, V′ is smaller than or equal to V, and W′ is larger than or equal to W; and
processing the X′ by Y′ by Z′ input tensor using the modified weight matrices to generate the U′ by V′ by W′ output tensor, wherein the U′ by V′ by W′ output tensor comprises the U by V by W output tensor.
19. A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations for training a convolutional neural network that includes a plurality of convolutional neural network layers on training data,
wherein the convolutional neural network comprises (i) at least one superpixel convolutional neural network layer having respective superpixel convolutional neural network layer parameters that is configured to receive an X by Y by Z input tensor and process the received input tensor using one or more convolutional neural network layer weight matrices to generate an U by V by W output tensor, and (ii) one or more other convolutional neural network layers having respective layer parameters, the operations comprising:
obtaining training data; and
training the convolutional neural network on the training data to adjust the values of the parameters of the superpixel convolutional neural network and the values of the parameters of the other convolutional neural network layers to trained values, and
wherein receiving, by the superpixel convolutional neural network layer, an X by Y by Z input tensor and processing the received input tensor using one or more convolutional neural network layer weight matrices to generate an U by V by W output tensor comprises:
transforming the X by Y by Z input tensor into a X′ by Y′ by Z′ input tensor, wherein X′ is smaller than or equal to X, Y′ is smaller than or equal to Y, and Z′ is larger than or equal to Z;
obtaining one or more modified convolutional neural network layer weight matrices, wherein (i) the modified weight matrices operate on the X′ by Y′ by Z′ input tensor to generate a U′ by V′ by W′ output tensor, and (ii) the U′ by V′ by W′ output tensor comprises a transformed U by V by W output tensor, wherein U′ is smaller than or equal to U, V′ is smaller than or equal to V, and W′ is larger than or equal to W; and
processing the X′ by Y′ by Z′ input tensor using the modified weight matrices to generate the U′ by V′ by W′ output tensor, wherein the U′ by V′ by W′ output tensor comprises the U by V by W output tensor.
20. The system of claim 18 , wherein X′ is equal to a ceiling of X divided by a natural number N, Y′ is equal to a ceiling of Y divided by a natural number M, and Z′ is equal to Z multiplied by NM, wherein the ceiling corresponds to a smallest integer larger or equal to.
21. The system of claim 20 , wherein the one or more modified weight matrices comprise NM copies of each original weight corresponding to the entries of the modified weight matrices.
22. The system of claim 21 , wherein training the convolutional neural network on the training data comprises performing a gradient descent and backpropagation neural network training process.
23. The system of claim 22 , wherein performing the backpropagation training process comprises adding weight gradients across all NM copies to determine a net change in value for each original weight.