Compiler for optimizing number of cores used to implement neural network
Some embodiments provide a compiler for optimizing the implementation of a machine-trained network (e.g., a neural network) on an integrated circuit (IC). The compiler of some embodiments receives a specification of a machine-trained network including multiple layers of computation nodes and generates a graph representing options for implementing the machine-trained network in the IC. In some embodiments, the graph includes nodes representing options for implementing each layer of the machine-trained network and edges between nodes for different layers representing different implementations that are compatible. In some embodiments, the graph is populated according to rules relating to memory use and the numbers of cores necessary to implement a particular layer of the machine trained network such that nodes for a particular layer, in some embodiments, represent fewer than all the possible groupings of sets of clusters.
1. A method for generating neural network program instructions for a neural network inference circuit (NNIC) to execute a neural network, the NNIC comprising a plurality of clusters of circuits for performing neural network computations, the method comprising:
receiving a specification of the neural network comprising a plurality of layers of computation nodes;
based on the received neural network specification, generating a graph comprising one or more options for executing each layer of the neural network on the NNIC, each respective option for a respective layer representing the use of a respective set of the clusters of the NNIC to execute the respective layer, wherein the graph comprises at least two options for at least one layer of the neural network;
identifying an optimized path through the graph that selects, for each layer of the neural network, one of the options to execute the layer of the neural network; and
generating the program instructions for the NNIC to use to execute the neural network based on the identified optimized path.
2. The method of claim 1 , wherein the at least two options for a particular layer of the neural network comprises a plurality of different options representing use of a same number of the clusters.
3. The method of claim 1 , wherein the options for a particular layer comprises at least two options representing use of different numbers of the clusters.
4. The method of claim 1 , wherein the graph is generated based on a set of rules for implementing neural networks using the NNIC and the options for a particular layer of the neural network excludes non-viable options for implementing the particular layer of the neural network using the NNIC.
5. The method of claim 4 , wherein the options for the particular layer excludes at least one viable option for implementing the particular layer of the neural network using the NNIC.
6. The method of claim 5 , wherein an excluded viable option for implementing the particular layer of the neural network represents the use of more clusters than are necessary to perform the processing for the particular layer of the neural network.
7. The method of claim 5 , wherein an excluded viable option for implementing the particular layer of the neural network includes an option that represents the use of non-adjacent clusters of the NNIC.
8. The method of claim 1 , wherein each cluster comprises one or more cores and each respective option for a respective layer further represents a respective number of cores of each cluster to use to execute the respective layer of the neural network.
9. The method of claim 8 , wherein the number of cores of each cluster to use is the same for each cluster in each respective option for implementing the respective layer of the neural network.
10. The method of claim 9 , wherein each core comprises corresponding pluralities of memory units for storing activations associated with layers of the neural network, wherein the program instructions include a specification of particular memory units to store input values associated with each layer of the neural network.
11. The method of claim 10 , wherein, for each layer of the neural network, the program instructions specify a same set of memory units in each core to store the activations associated with each layer of the neural network.
12. The method of claim 1 , wherein the optimized path through the graph is optimized for one of power, latency, memory use, and power at a particular latency.
13. The method of claim 1 , wherein the optimized path through the graph is a first optimized path and the method further comprises identifying a set of at least one additional optimized path through the graph, wherein each optimized path corresponds to a particular set of aspects of the neural network execution that is being optimized.
14. The method of claim 13 , wherein identifying an optimized path comprises generating and solving a mixed integer problem for optimizing a particular set of aspects of the neural network execution.
15. The method of claim 1 , wherein each cluster comprises a set of cores for computing partial dot products associated with convolutional layers of the neural network.
16. The method of claim 15 , wherein each core comprises a set of memory units storing (1) weight values for each layer implemented by the core and, while executing a particular layer of the neural network, (2) input values associated with the particular layer of the neural network.
17. The method of claim 16 , wherein each core further comprises a set of partial dot product computation circuits (PDPCCs) that compute dot products between weight values and input values of each layer of the neural network that the core implements.
18. The method of claim 1 , wherein different sets of clusters are identified to execute the adjacent layers of the neural network.
19. The method of claim 4 , wherein an option representing the use of a particular set of clusters of the NNIC for implementing a particular layer of the neural network is non-viable when there are too many non-zero weights in at least one set of filter channels in the particular layer that are assigned to a partial dot product computation circuit (PDPCC) of a particular core in the particular set of clusters for processing.
20. The method of claim 4 , wherein an option representing the use of a particular set of clusters of the NNIC for implementing a particular layer of the neural network is non-viable when there are too many dot-product terms in the particular layer to process using partial dot product computation circuits (PDPCCs) in the particular set of clusters.