Neural network inference circuit employing dynamic memory sleep
For a neural network inference circuit that executes a neural network including multiple computation nodes at multiple layers for which data is stored in a plurality of memory banks, some embodiments provide a method for dynamically putting memory banks into a sleep mode of operation to conserve power. The method tracks the accesses to individual memory banks and, if a certain number of clock cycles elapse with no access to a particular memory bank, sends a signal to the memory bank indicating that it should operate in a sleep mode. Circuit components involved in dynamic memory sleep, in some embodiments, include a core RAM pipeline, a core RAM sleep controller, a set of core RAM bank select decoders, and a set of core RAM memory bank wrappers.
1. For a sleep control circuit for a memory unit of a neural network inference circuit configured to execute a neural network, a method comprising:
determining a value indicating a length of time since the memory unit was last accessed, wherein the memory unit stores neural network weight data that is loaded onto the memory unit when the neural network inference circuit is booted up, the weight data for use each time the neural network inference circuit executes the neural network; and
when the determined value is at least a threshold value, sending a signal to the memory unit to operate in a sleep mode that is a power-saving mode of operation that maintains the neural network weight data stored in the memory unit.
2. The method of claim 1 , wherein the value is determined and the signal sent by a finite state machine of the sleep control circuit.
3. The method of claim 1 further comprising:
after sending the signal to the memory unit to operate in sleep mode, receiving a signal indicating that the memory unit has been selected for one of a read operation and a write operation; and
based on the signal indicating that the memory unit has been selected, sending a signal to the memory unit to operate in non-sleep mode.
4. The method of claim 3 further comprising resetting to zero the value indicating the length of time since the memory unit was last accessed.
5. The method of claim 1 , wherein the length of time is a number of clock cycles since the memory unit was last accessed.
6. The method of claim 1 , wherein the neural network inference circuit comprises:
a plurality of independently-controlled memory units; and
for each memory unit, a corresponding sleep control circuit.
7. The method of claim 6 , wherein each respective sleep control circuit maintains a respective value indicating a length of time since its corresponding memory unit was last accessed.
8. The method of claim 1 , wherein the memory unit comprises a first input for the signal to operate in sleep mode and a second input for a shutdown signal that indicates for the memory unit to power down.
9. The method of claim 8 , wherein the memory unit maintains data when operating in sleep mode but does not maintain data when powered down.
10. For a sleep control circuit for a memory unit of a neural network inference circuit configured to execute a neural network, a method comprising:
each clock cycle of the neural network inference circuit, determining a value indicating a number of clock cycles since the memory unit was last accessed; and
when the determined value is at least a threshold value, sending a signal to the memory unit to operate in a sleep mode.
11. The method of claim 10 , wherein each clock cycle, the sleep control circuit sends one of (i) the signal to operate in sleep mode and (ii) a signal to operate in non-sleep mode.
12. The method of claim 10 , wherein the sleep mode is a power-saving mode of operation that maintains data stored in the memory unit.
13. For a sleep control circuit for a memory unit of a neural network inference circuit configured to execute a neural network, a method comprising:
determining a value indicating a length of time since the memory unit was last accessed; and
when the determined value is at least a threshold value, sending a signal to the memory unit to operate in a sleep mode, wherein the neural network inference circuit comprises:
a plurality of independently-controlled memory units;
for each memory unit, a respective corresponding sleep control circuit that maintains a respective value indicating a length of time since its corresponding memory unit was last accessed; and
a memory unit selection decoder circuit that provides, to each of the sleep control circuits, a signal indicating whether the memory unit corresponding to the sleep control circuit is selected for access.
14. The method of claim 13 , wherein:
the plurality of memory units are part of a random-access memory (RAM) element of a dot product core of the neural network inference circuit; and
an access to the RAM element indicates (i) a particular one of the memory units to access and (ii) an address in the particular memory unit.
15. The method of claim 14 , wherein a signal indicating that the particular memory unit is selected is received at the sleep control circuit for the particular memory unit prior to the particular memory unit receiving input for the memory access.
16. The method of claim 15 , wherein the sleep control circuit for the particular memory unit sends a signal to the particular memory unit to operate in non-sleep mode prior to the particular memory unit receiving input for the memory access so that the particular memory unit has fully entered non-sleep mode when the input is received.
17. The method of claim 16 , wherein the neural network inference circuit comprises a set of registers that introduce latency into a memory access path.
18. For a sleep control circuit for a memory unit of a neural network inference circuit configured to execute a neural network, a method comprising:
determining a value indicating a length of time since the memory unit was last accessed; and
when the determined value is at least a threshold value, sending a signal to the memory unit to operate in a sleep mode,
wherein the neural network inference circuit comprises a plurality of independently-controlled memory units and, for each memory unit, a corresponding sleep control circuit,
wherein a compiler arranges for storage of weight and activation data in the plurality of memory units in order to conserve power by increasing an amount of time that the memory units operate in sleep mode.
19. The method of claim 18 , wherein the sleep mode is a power-saving mode of operation that maintains data stored in the memory unit.
20. The method of claim 19 , wherein the memory unit stores neural network weight data that is loaded onto the memory unit when the neural network inference circuit is booted up, the weight data for use each time the neural network inference circuit executes the neural network.