Multi-layer neural network system and method

1. A computer-implemented method comprising:

obtaining an input;

processing the input using a neural network comprising a plurality of layers, comprising:

calculating, using a first one or more layers of the plurality of layers, a first intermediate output from the input, wherein the first intermediate output is of a given size and is stored in memory;

reducing a size of one or more dimensions of the first intermediate output to generate a reduced first intermediate output that is smaller in size than the first intermediate output;

calculating, from the first intermediate output using a second one or more layers of the plurality of layers, a second intermediate output, the second one or more layers comprising one or more ultra-low precision layers;

deleting, after execution of at least one of the second one or more layers, the first intermediate output from the memory;

reducing a size of one or more dimensions of the second intermediate output to generate a reduced second intermediate output that is smaller in size than the second intermediate output;

calculating, after reducing the size of the one or more dimensions of the first intermediate output to generate the reduced first intermediate output and from the second intermediate output using a third one or more layers of the plurality of layers, a third intermediate output, wherein the first intermediate output is deleted from the memory prior to execution of at least one of the third one or more layers;

combining a plurality of reduced intermediate outputs to derive a combined intermediate output, wherein the plurality of reduced intermediate outputs combined comprise the reduced first intermediate output and the reduced second intermediate output; and

calculating, using one or more higher-precision layers of the plurality of layers, a neural network output using the combined intermediate output; and

outputting an output based on the neural network output.

2. The method of claim 1 , wherein reducing a size of one or more dimensions of the first intermediate output comprises reducing the spatial information of the first intermediate output and wherein reducing a size of one or more dimensions of the second intermediate output comprises reducing spatial information of the second intermediate output.

3. The method of claim 1 , wherein the first intermediate output is a higher-precision output and the second intermediate output is a higher-precision output.

4. The method of claim 1 , wherein reducing the spatial information of the first intermediate output comprises performing a pooling operation on the first intermediate output.

5. The method of claim 4 , wherein the pooling operation is a global pooling operation.

6. The method of claim 1 , wherein the one or more ultra-low precision layers comprise one or more binarized convolutional neural network layers.

7. The method of claim 1 , wherein the neural network comprises a first part comprising the layers which are executed prior to the combining step, the method further comprising:

for more than half of the convolutional layers in the first part of the neural network outputting a higher precision intermediate output, reducing a size of one or more dimensions of the respective intermediate output to give a respective reduced intermediate output;

wherein the plurality of reduced intermediate outputs comprises the respective reduced intermediate outputs.

8. The method of claim 1 , wherein a neural network layer receiving the input is a higher precision layer.

9. The method of claim 8 , wherein the neural network layer receiving the input is a convolutional neural network layer which reduces size of one or more spatial dimensions, and wherein the subsequent neural network layer is a higher precision convolutional neural network layer which increases the number of channels.

10. The method of claim 1 , wherein the combined intermediate output is used to weight the output channels of a convolutional neural network layer.

11. A computer-implemented method for training a neural network comprising a plurality of layers, the method comprising:

performing a forward pass using the neural network for each training sample of one or more training samples, each training sample comprising an input, performing each forward pass comprising:

calculating, using a first one or more layers of the plurality of layers, a first intermediate output from the input, wherein the first intermediate output is of a given size and is stored in memory;

reducing a size of one or more dimensions of the first intermediate output to generate a reduced first intermediate output that is smaller in size than the first intermediate output;

calculating, from the first intermediate output using a second one or more layers of the plurality of layers, a second intermediate output, the second one or more layers comprising one or more ultra-low precision layers;

reducing a size of one or more dimensions of the second intermediate output to generate a reduced second intermediate output that is smaller in size than the second intermediate output;

deleting, after execution of at least one of the second one or more layers, the first intermediate output from the memory;

calculating, after reducing the size of the one or more dimensions of the first intermediate output to generate the reduced first intermediate output and from the second intermediate output using a third one or more layers of the plurality of layers, a third intermediate output, wherein the first intermediate output is deleted from the memory prior to execution of at least one of the third one or more layers;

combining a plurality of reduced intermediate outputs to derive a combined intermediate output, wherein the plurality of reduced intermediate outputs combined comprise the reduced first intermediate output and the reduced second intermediate output; and

calculating, using one or more higher-precision layers of the plurality of layers, a neural network output using the combined intermediate output; and

updating the parameters of the neural network.

12. A system comprising one or more processors, the one or more processors configured to:

process an input using a neural network comprising a plurality of layers, comprising:

calculating, using a first one or more layers of the plurality of layers, a first intermediate output from the input, wherein the first intermediate output is of a given size and is stored in memory;

reducing a size of one or more dimensions of the first intermediate output to generate a reduced first intermediate output that is smaller in size than the first intermediate output;

calculating, from the first intermediate output using a second one or more layers of the plurality of layers, a second intermediate output, the second one or more layers comprising one or more ultra-low precision layers;

reducing a size of one or more dimensions of the second intermediate output to generate a reduced second intermediate output that is smaller in size than the second intermediate output;

deleting, after execution of at least one of the second one or more layers, the first intermediate output from the memory;

calculating, after reducing the size of the one or more dimensions of the first intermediate output to generate the reduced first intermediate output and from the second intermediate output using a third one or more layers of the plurality of layers, a third intermediate output, wherein the first intermediate output is deleted from the memory prior to execution of at least one of the third one or more layers;

combining a plurality of reduced intermediate outputs to derive a combined intermediate output, wherein the plurality of reduced intermediate outputs combined comprise the reduced first intermediate output and the reduced second intermediate output; and

calculating, using one or more higher-precision layers of the plurality of layers, a neural network output using the combined intermediate output.

13. The system of claim 12 , wherein reducing a size of one or more dimensions of the first intermediate output comprises reducing the spatial information of the first intermediate output and wherein reducing a size of one or more dimensions of the second intermediate output comprises reducing spatial information of the second intermediate output.

14. The system of claim 12 , wherein reducing the spatial information of the first intermediate output comprises performing a pooling operation on the first intermediate output.

15. The system of claim 14 , wherein the pooling operation is a global pooling operation.

16. The system of claim 12 , wherein the neural network comprises a first part comprising the layers which are executed prior to the combining step, wherein process an input using a neural network further comprises:

for more than half of the convolutional layers in the first part of the neural network outputting a higher precision intermediate output, reducing a size of one or more dimensions of the respective intermediate output to give a respective reduced intermediate output;

wherein the plurality of reduced intermediate outputs comprises the respective reduced intermediate outputs.

17. The system of claim 12 , wherein a neural network layer receiving the input is a higher precision layer.

18. The system of claim 12 , wherein the neural network layer receiving the input is a convolutional neural network layer which reduces size of one or more spatial dimensions, and wherein the subsequent neural network layer is a higher precision convolutional neural network layer which increases the number of channels.

19. The method of claim 1 , wherein the input comprises image data, and wherein the method further comprises determining that at least one object is present in the image based on the neural network output, and outputting an output indicating that the at least one object is present in the image.

20. The method of claim 1 ,

wherein the first intermediate output comprises a 3D matrix of values, the 3D matrix of values having a first height, a first width and a first number of channels, wherein each channel comprises a 2D matrix of values having the first height and the first width, and

wherein reducing the size of one or more dimensions of the first intermediate output to generate the reduced first intermediate output comprises deriving a single value for each of a plurality of areas of the 2D matrices to generate a 3D matrix having a second height, a second width and the first number of channels, wherein the second height is smaller than the first height and the second width is smaller than the first width.

21. The method of claim 1 ,

wherein the first intermediate output comprises a 3D matrix of values, the 3D matrix of values having a first height, a first width and a first number of channels, wherein each channel comprises a 2D matrix of values having the first height and the first width, and

wherein reducing the size of one or more dimensions of the first intermediate output to generate the reduced first intermediate output comprises deriving a single value for each channel to generate a vector comprising a single value for each channel.

22. The method of claim 1 , further comprising deleting the reduced first intermediate output from memory after the combining of plurality of reduced intermediate outputs to derive the combined intermediate output.