Compiler for implementing memory shutdown for neural network implementation configuration
Some embodiments provide a compiler for optimizing the implementation of a machine-trained network (e.g., a neural network) on an integrated circuit (IC). The compiler of some embodiments receives a specification of a machine-trained network including multiple layers of computation nodes and generates a graph representing options for implementing the machine-trained network in the IC. In some embodiments, the graph includes nodes representing options for implementing each layer of the machine-trained network and edges between nodes for different layers representing different implementations that are compatible. The compiler of some embodiments is also responsible for generating instructions relating to shutting down (and waking up) memory units of cores. In some embodiments, the memory units to shutdown are determined by the compiler based on the data that is stored or will be stored in the particular memory units.
1. A method for generating neural network program instructions for a neural network inference circuit (NNIC) to execute a neural network comprising a plurality of layers, the method comprising:
receiving configuration data for executing the neural network on the NNIC, the configuration data comprising data specifying which memory units of the NNIC are used to execute each layer of the neural network;
based on the received configuration data, identifying, for each layer of the neural network, (1) a first set of memory units for which being shut down during execution of the layer saves power and does not affect execution of subsequent layers and (2) a second set of memory units to be powered on during execution of the layer in order to be available during execution of a subsequent layer; and
generating a set of program instructions for the NNIC to use to execute the particular neural network, wherein the set of program instructions comprises, for each layer of the neural network, a set of program instructions for (1) shutting down active memory units in the first set of memory units for the layer and (2) powering on the memory units of the second set of memory units during the execution of the layer.
2. The method of claim 1 , wherein the first set of memory units for a particular layer of the neural network comprises less than all the memory units not used in executing the particular layer.
3. The method of claim 1 , wherein the first set of memory units for a particular layer of the neural network comprises no memory units.
4. The method of claim 1 , wherein:
the NNIC comprises a plurality of identical clusters identified by different cluster indices;
each cluster comprises a plurality of identical cores identified by an identical plurality of core indices within each cluster;
each core comprises an identical plurality of memory units identified by an identical plurality of memory unit indices within each core; and
for at least one layer of the neural network, the first set of memory units comprises memory units on each core executing the at least one layer identified by a same set of memory unit indices.
5. The method of claim 1 , wherein:
the NNIC comprises a plurality of identical clusters identified by different cluster indices;
each cluster comprises a plurality of identical cores identified by an identical plurality of core indices within each cluster;
each core comprises an identical plurality of memory units identified by an identical plurality of memory unit indices within each core; and
for at least one layer of the neural network, the first set of memory units comprises third and fourth sets of memory units of first and second cores, respectively, executing the at least one layer, wherein the memory units of the third set are identified by a different set of memory unit indices than are used to identify memory units of the fourth set.
6. The method of claim 1 further comprising identifying a time in the execution of the layer that each memory unit of the second set of memory units should be powered on, wherein the program instructions for powering on the second set of memory units comprise program instructions for powering on each memory unit in the second set of memory units at the identified time.
7. The method of claim 6 , wherein a particular number of cycles pass between initiating powering on of a memory unit and the memory unit being available for processing, wherein the particular number of cycles is used to identify the time in the execution of the layer that each memory unit of the second set of memory units should be powered on.
8. The method of claim 1 , wherein the first set of memory units comprises at least one memory unit that is shut down at the beginning of the execution of the layer.
9. The method of claim 1 , wherein (i) shutting down a memory unit saves power when the memory unit is not configured to be accessed for a particular number of clock cycles, (ii) shutting down a memory unit causes data stored in the memory unit to be lost, and (iii) identifying the first set of memory units comprises:
identifying a third set of memory units for which being shut down does not affect the execution of the layer; and
identifying the first set of memory units by excluding from the third set of memory units at least (i) memory units that are not accessed for a number of clock cycles less than the particular number of clock cycles and (ii) memory units storing an input value that is used during the execution of at least one subsequent layer.
10. The method of claim 1 , wherein a power cost is associated with shutting down a memory unit, wherein shutting down a memory unit saves power when the power saved by the memory unit being shutdown is greater than the power cost of shutting down the memory unit.
11. A non-transitory machine readable medium storing a program for execution by a set of processing units, the program for generating neural network program instructions for a neural network inference circuit (NNIC) to execute a neural network comprising a plurality of layers, the program comprising sets of instructions for:
receiving configuration data for executing the neural network on the NNIC, the configuration data comprising data specifying which memory units of the NNIC are used to execute each layer of the neural network;
based on the received configuration data, identifying, for each layer of the neural network, (1) a first set of memory units for which being shut down during execution of the layer saves power and does not affect execution of subsequent layers and (2) a second set of memory units to be powered on during execution of the layer in order to be available during execution of a subsequent layer; and
generating a set of program instructions for the NNIC to use to execute the particular neural network, wherein the set of program instructions comprises, for each layer of the neural network, a set of program instructions for (1) shutting down active memory units in the first set of memory units for the layer and (2) powering on the memory units of the second set of memory units during the execution of the layer.
12. The non-transitory machine readable medium of claim 11 , wherein the first set of memory units for a particular layer of the neural network comprises less than all the memory units not used in executing the particular layer.
13. The non-transitory machine readable medium of claim 11 , wherein the first set of memory units for a particular layer of the neural network comprises no memory units.
14. The non-transitory machine readable medium of claim 11 , wherein:
the NNIC comprises a plurality of identical clusters identified by different cluster indices;
each cluster comprises a plurality of identical cores identified by an identical plurality of core indices within each cluster;
each core comprises an identical plurality of memory units identified by an identical plurality of memory unit indices within each core; and
for at least one layer of the neural network, the first set of memory units comprises memory units on each core executing the at least one layer identified by a same set of memory unit indices.
15. The non-transitory machine readable medium of claim 11 , wherein:
the NNIC comprises a plurality of identical clusters identified by different cluster indices
each cluster comprises a plurality of identical cores identified by an identical plurality of core indices within each cluster;
each core comprises an identical plurality of memory units identified by an identical plurality of memory unit indices within each core; and
for at least one layer of the neural network, the first set of memory units comprises third and fourth sets of memory units of first and second cores, respectively, executing the at least one layer, wherein the memory units of the third set are identified by a different set of memory unit indices than are used to identify memory units of the fourth set.
16. The non-transitory machine readable medium of claim 11 , wherein the program further comprises a set of instructions for identifying a time in the execution of the layer that each memory unit of the second set of memory units should be powered on, wherein the generated program instructions for powering on the second set of memory units comprise generated program instructions for powering on each memory unit in the second set of memory units at the identified time.
17. The non-transitory machine readable medium of claim 16 , wherein a particular number of cycles pass between initiating powering on of a memory unit and the memory unit being available for processing, wherein the particular number of cycles is used to identify the time in the execution of the layer that each memory unit of the second set of memory units should be powered on.
18. The non-transitory machine readable medium of claim 11 , wherein the first set of memory units comprises at least one memory unit that is shut down at the beginning of the execution of the layer.
19. The non-transitory machine readable medium of claim 11 , wherein (i) shutting down a memory unit saves power when the memory unit is not configured to be accessed for a particular number of clock cycles, (ii) shutting down a memory unit causes data stored in the memory unit to be lost, and (iii) the set of instructions for identifying the first set of memory units comprises:
identifying a third set of memory units for which being shut down does not affect the execution of the layer; and
identifying the first set of memory units by excluding from the third set of memory units at least (i) memory units that are not accessed for a number of clock cycles less than the particular number of clock cycles and (ii) memory units storing an input value that is used during the execution of at least one subsequent layer.
20. The non-transitory machine readable medium of claim 11 , wherein a power cost is associated with shutting down a memory unit, wherein shutting down a memory unit saves power when the power saved by the memory unit being shutdown is greater than the power cost of shutting down the memory unit.