Compiler for implementing gating functions for neural network configuration
Some embodiments provide a compiler for optimizing the implementation of a machine-trained network (e.g., a neural network) on an integrated circuit (IC). The compiler of some embodiments receives a specification of a machine-trained network including multiple layers of computation nodes and generates a graph representing options for implementing the machine-trained network in the IC. In some embodiments, the compiler also generates instructions for gating operations. Gating operations, in some embodiments, include gating at multiple levels (e.g., gating of clusters, cores, or memory units). Gating operations conserve power in some embodiments by gating signals so that they do not reach the gated element or so that they are not propagated within the gated element. In some embodiments, a clock signal is gated such that a register that transmits data on a rising (or falling) edge of a clock signal is not triggered.
1. A method for generating neural network program instructions for a neural network inference circuit (NNIC) to execute a neural network, the method comprising:
receiving a specification of the neural network comprising a plurality of layers of computation nodes; and
based on the received neural network specification, identifying elements of the NNIC that are not needed to execute each layer of a set of the layers, the NNIC elements comprising a plurality of clusters, each cluster comprising a set of dot product cores for (i) storing weight values and intermediate activation values during execution of the neural network and (ii) performing dot product computations between the weight values and the intermediate activation values; and
based on the received specification and the identified elements, generating program instructions for the NNIC to use to execute the neural network, the program instructions specifying gated elements for each layer of the set of layers in order to reduce power usage by the NNIC when executing the neural network, the gated elements for a first layer comprising at least one cluster of dot product cores that are gated during the execution of the first layer and not gated during the execution of a second layer.
2. The method of claim 1 , wherein an element gated during a particular layer does not propagate data during execution of the particular layer.
3. The method of claim 1 , wherein each dot product core comprises a set of memory units that store weight values and intermediate activation values.
4. The method of claim 1 , wherein the program instructions specify that a particular dot product core is gated for a third layer but that the cluster to which the particular dot product core belongs is not gated for the third layer.
5. The method of claim 3 , wherein the program instructions specify that a particular memory unit is gated for a third layer but that the dot product core to which the particular memory unit belongs is not gated for the third layer.
6. The method of claim 1 , wherein:
the received specification is used to determine an optimized implementation for the neural network using the NNIC;
the optimized implementation identifies a set of elements used to execute each layer of the neural network; and
identifying elements of the neural network inference circuit that are not needed to execute the layer comprises using the optimized implementation to identify the elements that are not needed to execute the layer.
7. The method of claim 3 , wherein:
when the program instructions specify to gate a particular cluster, the dot product cores belonging to the particular cluster and the memory units belonging to the dot product cores of the particular cluster are not separately gated; and
when the program instructions specify to gate a particular dot product core, the memory units of the particular dot product core are not separately gated.
8. The method of claim 1 , wherein gating an element comprises gating a clock signal.
9. The method of claim 8 , wherein gating the clock signal prevents the propagation of data by a set of registers triggered by the clock signal.
10. A non-transitory machine readable medium storing a program for execution by a set of processing units, the program for generating neural network program instructions for a neural network inference circuit (NNIC) to execute a neural network, the program comprising sets of instructions for:
receiving a specification of the neural network comprising a plurality of layers of computation nodes; and
based on the received neural network specification, identifying elements of the NNIC that are not needed to execute each layer of a set of the layers, the NNIC elements comprising a plurality of clusters, each cluster comprising a set of dot product cores for (i) storing weight values and intermediate activation values during execution of the neural network and (ii) performing dot product computations between the weight values and the intermediate activation values; and
based on the received specification and the identified elements, generating program instructions for the NNIC to use to execute the neural network, the program instructions specifying gated elements for each layer of the set of layers in order to reduce power usage by the NNIC when executing the neural network, the gated elements for a first layer comprising at least one cluster of dot product cores that are gated during the execution of the first layer and not gated during the execution of a second layer.
11. The non-transitory machine readable medium of claim 10 , wherein an element gated during a particular layer does not propagate data during execution of the particular layer.
12. The non-transitory machine readable medium of claim 10 , wherein each dot product core comprises a set of memory units that store weight values and intermediate activation values.
13. The non-transitory machine readable medium of claim 10 , wherein the program instructions specify that a particular dot product core is gated for a third layer but that the cluster to which the particular dot product core belongs is not gated for the third layer.
14. The non-transitory machine readable medium of claim 12 , wherein the program instructions specify that a particular memory unit is gated for a third layer but that the dot product core to which the particular memory unit belongs is not gated for the third layer.
15. The non-transitory machine readable medium of claim 10 , wherein:
the received specification is used to determine an optimized implementation for the neural network using the NNIC;
the optimized implementation identifies a set of elements used to execute each layer of the neural network; and
the set of instructions for identifying elements of the neural network inference circuit that are not needed to execute the layer comprises a set of instructions for using the optimized implementation to identify the elements that are not needed to execute the layer.
16. The non-transitory machine readable medium of claim 12 , wherein:
when the program instructions specify to gate a particular cluster, the dot product cores belonging to the particular cluster and the memory units belonging to the dot product cores of the particular cluster are not separately gated; and
when the program instructions specify to gate a particular dot product core, the memory units of the particular dot product core are not separately gated.
17. The non-transitory machine readable medium of claim 10 , wherein gating an element comprises gating a clock signal.
18. The non-transitory machine readable medium of claim 17 , wherein gating the clock signal prevents the propagation of data by a set of registers triggered by the clock signal.
19. The method of claim 1 , wherein the dot product cores of the cluster gated during the execution of the first layer do not perform any dot product operations during the first layer.
20. The method of claim 1 , wherein dot products computed during the first layer are small enough that not all of the clusters of the NNIC are required to be used during the first layer.