Systems and methods for improved neural network execution

1. A method for computing one or more outputs of a neural network having a plurality of layers, the method comprising:

analyzing, by a computing device, the neural network to determine a plurality of sub-computations from total computations of the neural network, wherein determining each sub-computation further comprises:

i) determining a group of outputs for each sub-computation based on one layer of the plurality of layers;

ii) for each group of outputs, determining a largest number of inputs in some layer that precedes the one layer that are necessary to compute the respective group of outputs and results in a memory requirement less than a memory threshold,

wherein at least two of the largest number of inputs for the respective sub-computation overlap and span at least two layers of the plurality of layers of the neural network; and

computing the outputs of the neural network by executing each sub-computation.

2. The method of claim 1 wherein executing each sub-computation further comprises identifying a set of corresponding cores that together have a cache memory size that is at least the memory threshold.

3. The method of claim 2 wherein the cache memory size is a L 2 or L 3 cache memory size.

4. The method of claim 2 wherein the cache memory size is total cache size needed by the neural network to execute the computations of any intermediate layer of the respective sub-computations of the neural network.

5. The method of claim 4 wherein determining the largest number of inputs in some preceding layer that are necessary to compute the respective group of outputs and results in a memory requirement less than a memory threshold further comprises:

receiving a bottom layer;

adding all outputs in each layer neural network from the start layer traversing the bottom layer that contribute to computing the one previously unselected output into the group of outputs; and

comparing the group of outputs to the memory threshold.

6. The method of claim 1 wherein determining a group of outputs for each sub-computation further comprises receiving a start layer and selecting one previously unselected output from the start layer to add to the group of outputs.

7. The method of claim 1 further comprising:

determining each repeated computation in the plurality of sub-computations;

executing each repeated computation only once;

storing a result of each repeated computation; and

reusing the result of each repeated computation for each sub-computation of the plurality of sub-computations that has not yet executed and has a computation that is the same as any of the repeated computations with stored results.

8. The method of claim 7 wherein at least some of the each repeated computations in the plurality of sub-computations is stored on a shared memory multiprocessor and accessed by multiple computing cores.

9. The method of claim 1 wherein the plurality of sub-computations are grouped into workspaces and wherein each workspace is assigned a set of cores to execute on.

10. The method of claim 1 further comprising determining whether to avoid computing one or more computations that repeat in the plurality of computations based on a repeat ratio, wherein the repeat ratio is based on the number of computations in the neural network and the number of computations that are repeat computations.

11. The method of claim 1 wherein computing the outputs of the neural network by executing each sub-computation further comprises writing outputs from computations other than a final output layer of the neural network into cache only.

12. The method of claim 1 wherein computing the outputs of the neural network by executing each sub-computation further comprises reading outputs from computations other than a final output layer of the neural network from cache only.

13. The method of claim 1 wherein the neural network is a convolutional neural network.

14. The method of claim 1 wherein the plurality of layers is any combination of a direct convolutional layer, Fast Fourier Transform layer, Winograd Transform layer, any pooling layer, any non-element-wise layer, element-wise-layer, and any Fully-Connected layer.

15. The method of claim 1 wherein the plurality of layers is a sparsification of an original network layer.

16. A system comprising an improved convolutional neural network (CNN), the CNN comprising:

a memory; and

a processor; the processor configured to:

analyzing, by a computing device, the neural network to determine a plurality of sub-computations from total computations of the neural network, wherein determining each sub-computation further comprises:

iii) determining a group of outputs for each sub-computation based on one layer of the plurality of layers;

iv) for each group of outputs, determining a largest number of inputs in some layer that precedes the one layer that are necessary to compute the respective group of outputs and results in a memory requirement less than a memory threshold,

wherein at least two of the largest number of inputs for the respective sub-computation overlap and span at least two layers of the plurality of layers of the neural network; and

computing the outputs of the neural network by executing each sub-computation.

17. The system of claim 16 further comprising:

determining each repeated computation in the plurality of sub-computations;

executing each repeated computation only once;

storing a result of each repeated computation; and

reusing the result of each repeated computation for each sub-computation of the plurality of sub-computations that has not yet executed and has a computation that is the same as any of the repeated computations with stored results.

18. The system of claim 17 wherein at least some of the each repeated computations in the plurality of sub-computations is stored on a shared memory multiprocessor and accessed by multiple computing cores.

19. A non-transitory computer readable storage medium storing a set of instructions for causing a computer to generate a performance rating for a developer, by performing the operations of:

analyzing the neural network to determine a plurality of sub-computations from total computations of the neural network, wherein determining each sub-computation further comprises:

v) determining a group of outputs for each sub-computation based on one layer of the plurality of layers;

vi) for each group of outputs, determining a largest number of inputs in some layer that precedes the one layer that are necessary to compute the respective group of outputs and results in a memory requirement less than a memory threshold,

wherein at least two of the largest number of inputs for the respective sub-computation overlap and span at least two layers of the plurality of layers of the neural network; and

computing the outputs of the neural network by executing each sub-computation.

20. The non-transitory computer readable storage medium of claim 19 wherein executing each sub-computation further comprises identifying a set of corresponding cores that together have a cache memory size that is at least the memory threshold.