Allocating blocks of unified memory for integrated circuit executing neural network
Some embodiments provide a method that receives a specification of a neural network for execution by an integrated circuit. The integrated circuit includes a neural network inference circuit for executing the neural network to generate an output based on an input, an input processing circuit for providing the input to the neural network inference circuit, a microprocessor circuit for controlling the neural network inference circuit and the input processing circuit, and a unified memory accessible by the microprocessor circuit, the neural network inference circuit, and the input processing circuit. The method determines usage of the unified memory by the neural network inference circuit while executing the neural network. Based on the determined usage by the neural network inference circuit, the method allocates portions of the unified memory to the microprocessor circuit and input processing circuit.
1. A method for a compiler that generates configuration data to enable an integrated circuit (IC) to execute a neural network, the method comprising:
receiving a specification of the neural network for execution by the integrated circuit, the integrated circuit comprising (i) a neural network inference circuit for executing the neural network to generate an output based on an input, (ii) an input processing circuit for providing the input to the neural network inference circuit, (iii) a microprocessor circuit for controlling the neural network inference circuit and the input processing circuit, and (iv) a unified memory accessible by the microprocessor circuit, the neural network inference circuit, and the input processing circuit;
determining portions of the unified memory for the neural network inference circuit to use when the neural network inference circuit executes the neural network; and
based on the determined portions of the unified memory that the neural network inference circuit will use when executing the neural network, allocating other portions of the unified memory to the microprocessor circuit and input processing circuit.
2. The method of claim 1 , wherein the neural network inference circuit stores weight values and activation values in the unified memory.
3. The method of claim 2 , wherein the amount of unified memory required by the neural network inference circuit varies during execution of the neural network based on a number of activation values stored in the unified memory.
4. The method of claim 1 , wherein the integrated circuit (i) is incorporated into a device that comprises a sensor, (ii) receives inputs from the sensor, and (iii) stores the inputs in the unified memory in order for the neural network inference circuit to generate outputs based on the inputs according to instructions from the input processing circuit.
5. The method of claim 1 , wherein the unified memory is organized as a plurality of memory banks.
6. The method of claim 5 , wherein determining portions of the unified memory for the neural network inference circuit to use comprises generating neural network program instructions that specify memory banks used by the neural network inference circuit for storing weight values and activation values at each of a plurality of layers of the neural network.
7. The method of claim 6 , wherein allocating portions of the unified memory to the microprocessor circuit and input processing circuit comprises:
based on a size of the inputs to be provided to the neural network inference circuit and a number of the inputs to be stored in the unified memory at the same time, allocating memory banks to the input processing circuit for storing the inputs; and
allocating a set of memory banks to the microprocessor circuit that are not already allocated to the neural network inference circuit or the input processing circuit.
8. The method of claim 5 , wherein:
the neural network inference circuit comprises a plurality of cores;
each respective core is associated with a respective set of the memory banks of the unified memory such that each memory bank is associated with one of the cores and no memory bank is associated with multiple cores; and
different cores use different numbers of memory banks when the neural network inference circuit executes the neural network.
9. The method of claim 8 , wherein the memory banks that are not used by the cores of the neural network inference circuit are organized into a single set of memory banks for a virtual addressing scheme.
10. The method of claim 9 , wherein the virtual addressing scheme is used by the microprocessor circuit and the input processing circuit when addressing the unified memory.
11. A non-transitory machine-readable medium storing a program for execution by at least one processing unit, the program comprising sets of instructions for:
receiving a specification of a neural network for execution by an integrated circuit that comprises a neural network inference circuit for executing the neural network to generate an output based on an input, an input processing circuit for providing the input to the neural network inference circuit, a microprocessor circuit for controlling the neural network inference circuit and the input processing circuit, and a unified memory accessible by the microprocessor circuit, the neural network inference circuit, and the input processing circuit;
determining usage of the unified memory by the neural network inference circuit while executing the neural network; and
based on the determined usage by the neural network inference circuit, allocating portions of the unified memory to the microprocessor circuit and input processing circuit.
12. The non-transitory machine-readable medium of claim 11 , wherein the neural network inference circuit stores weight values and activation values in the unified memory.
13. The non-transitory machine-readable medium of claim 12 , wherein the amount of unified memory required by the neural network inference circuit varies during execution of the neural network based on a number of activation values stored in the unified memory.
14. The non-transitory machine-readable medium of claim 11 , wherein the integrated circuit (i) is incorporated into a device that comprises a sensor, (ii) receives inputs from the sensor, and (iii) stores the inputs in the unified memory in order for the neural network inference circuit to generate outputs based on the inputs according to instructions from the input processing circuit.
15. The non-transitory machine-readable medium of claim 11 , wherein the unified memory is organized as a plurality of memory banks.
16. The non-transitory machine-readable medium of claim 15 , wherein the set of instructions for determining usage of the unified memory by the neural network inference circuit comprises a set of instructions for generating neural network program instructions that specify memory banks used by the neural network inference circuit for storing weight values and activation values at each of a plurality of layers of the neural network.
17. The non-transitory machine-readable medium of claim 16 , wherein the set of instructions for allocating portions of the unified memory to the microprocessor circuit and input processing circuit comprises sets of instructions for:
based on a size of the inputs to be provided to the neural network inference circuit and a number of the inputs to be stored in the unified memory at a given time, allocating memory banks to the input processing circuit for storing the inputs; and
allocating a set of memory banks to the microprocessor circuit that are not already allocated to the neural network inference circuit or the input processing circuit.
18. The non-transitory machine-readable medium of claim 15 , wherein:
the neural network inference circuit comprises a plurality of cores;
each respective core is associated with a respective set of the memory banks of the unified memory such that each memory bank is associated with one of the cores and no memory bank is associated with multiple cores; and
different cores use different numbers of memory banks when the neural network inference circuit executes the neural network.
19. The non-transitory machine-readable medium of claim 18 , wherein the memory banks that are not used by the cores of the neural network inference circuit are organized into a single set of memory banks for a virtual addressing scheme.
20. The non-transitory machine-readable medium of claim 19 , wherein the virtual addressing scheme is used by the microprocessor circuit and the input processing circuit when addressing the unified memory.