Scheduler for mapping neural networks onto an array of neural cores in an inference processing unit
Mapping of neural network layers to physical neural cores is provided. In various embodiments, a neural network description describing a plurality of neural network layers is read. Each of the plurality of neural network layers has an associated weight tensor, input tensor, and output tensor. A plurality of precedence relationships among the plurality of neural network layers is determined. The weight tensor, input tensor, and output tensor of each of the plurality of neural network layers are mapped onto an array of neural cores.
1. A method comprising:
reading a neural network description describing a plurality of neural network layers, each of the plurality of neural network layers having an associated parameter tensor, weight tensor, input tensor, and output tensor;
determining a plurality of precedence relationships among the plurality of neural network layers and a plurality of logical cores, wherein each logical core corresponds to a neural network layer, wherein the precedence relationships are based on connections among the plurality of neural network layers;
based on the plurality of precedence relationships, generating a sequence of the plurality of neural network layers and a sequence of the plurality of logical cores;
identifying a set of identical logical cores from the plurality of logical cores, wherein the identical logical cores correspond to identical neural network layers, wherein the identical neural network layers are associated with identical parameter tensors and identical weight tensors; and
mapping the plurality of logical cores onto an array of neural cores such that the identical logical cores are mapped onto a same neural core, wherein the mapping of the plurality of logical cores on the array of neural cores comprises:
determining a plurality of sequence numbers, each sequence number of the plurality of sequence numbers corresponding to one neural core of the array of neural cores,
mapping the associated parameter tensor, weight tensor, input tensor, and output tensor of each of the plurality of neural network layers onto the array of neural cores, and
determining a schedule for delivery of the weight tensor, the input tensor, and the output tensor of each of the plurality of neural network layers in accordance with the plurality of sequence numbers, the data delivery being at each of the plurality of neural cores for computation of each neural network layer of the plurality of neural network layers.
2. The method of claim 1 , wherein the mapping of the weight tensor, the input tensor, and the output tensor comprises computation and communication operations.
3. The method of claim 2 , further comprising:
generating microcode, the microcode executable by a chip to compute the plurality of neural network layers, the chip comprising the array of neural cores.
4. The method of claim 3 , further comprising:
generating core microcode, the core microcode executable by the array of neural cores to generate partial sums.
5. The method of claim 3 , further comprising:
generating chip microcode, the chip microcode executable by at least one chip microengine to distribute weights, parameters, instructions, and/or activation data to the array of neural cores.
6. The method of claim 1 , wherein the mapping of the weight tensor, input tensor, and output tensor comprises determining a memory allocation for the weight tensor, input tensor, and output tensor of each of the plurality of neural network layers.
7. The method of claim 6 , wherein the mapping of the weight tensor, input tensor, and output tensor comprises: determining a memory allocation for plurality of partial sums of each of the plurality of neural network layers.
8. The method of claim 6 , wherein the mapping of the weight tensor, input tensor, and output tensor comprises: determining a memory allocation for the weight tensor in a global memory.
9. The method of claim 8 , wherein the mapping of the weight tensor, input tensor, and output tensor comprises: determining a plurality of mapping functions of the weight tensor to local memories of the neural cores.
10. The method of claim 6 , wherein the mapping of the weight tensor, input tensor, and output tensor comprises: determining a memory allocation for the input tensor and output tensor in local memories of the neural cores.
11. The method of claim 10 , wherein the memory allocation minimizes a transfer of activations among cores.
12. The method of claim 10 , wherein determining the memory allocation comprises sequencing the plurality of neural network layers to conform to a local memory limit.
13. The method of claim 10 , wherein the schedule for the data delivery is determined such that the memory allocation is applied when the execution requires the input tensor and output tensor.
14. The method of claim 13 , wherein the local memories are reallocated repeatedly according to the schedule for the data delivery.
15. The method of claim 1 , wherein each layer of the neural network has an associated tensor operation, the method further comprising:
selecting a parametrized scheme from a library of parametrized schemes, the selected scheme corresponding to the tensor operation, wherein:
the mapping comprises instantiating the selected scheme with parameters corresponding to the tensor operation.
16. The method of claim 15 , wherein the mapping comprises determining an execution schedule comprising a plurality of operations for the array of neural cores, the execution schedule guaranteeing data delivery at each of the plurality of neural cores for the computation of each neural network layer.
17. The method of claim 16 , wherein the execution schedule comprises computation and communication operations.
18. The method of claim 17 , wherein the execution schedule further comprises no operations (NOPs).
19. The method of claim 1 , wherein the mapping comprises determining a network schedule, wherein the network schedule determines a timing of communications on one or more networks.
20. The method of claim 19 , wherein the one or more networks comprises a weight network, an instruction network, an activation network, or a partial sum network.
21. The method of claim 20 , wherein the network schedule precludes simultaneous transmission on each network.
22. The method of claim 19 , wherein the network schedule guarantees data delivery at each of the plurality of neural cores for each computation using said data.
23. The method of claim 1 , wherein the mapping comprises determining a batch size.
24. The method of claim 23 , wherein the batch size is unique for at least one neural network layer.
25. A system comprising:
a chip comprising a controller and an array of neural cores;
a computing node comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor of the computing node to cause the processor to perform a method comprising:
reading a neural network description describing a plurality of neural network layers, each of the plurality of neural network layers having an associated parameter tensor, weight tensor, input tensor, and output tensor;
determining a plurality of precedence relationships among the plurality of neural network layers and a plurality of logical cores, wherein each logical core corresponds to a neural network layer, wherein the precedence relationships are based on connections among the plurality of neural network layers;
based on the plurality of precedence relationships, generating a sequence of the plurality of neural network layers and a sequence of the plurality of logical cores;
identifying a set of identical logical cores from the plurality of logical cores, wherein the identical logical cores correspond to identical layers, wherein the identical layers are associated with identical parameter tensors and identical weight tensors;
mapping the plurality of logical cores onto an array of neural cores such that the identical logical cores are mapped onto a same neural core, wherein the mapping of the plurality of logical cores on the array of neural cores comprises:
determining a plurality of sequence numbers, each sequence number of the plurality of sequence numbers corresponding to one neural core of the array of neural cores,
mapping the associated parameter tensor, weight tensor, input tensor, and output tensor of each of the plurality of neural network layers onto the array of neural cores, and
determining a schedule for delivery of the weight tensor, the input tensor, and the output tensor of each of the plurality of neural network layers in accordance with the plurality of sequence numbers, the data delivery being at each of the plurality of neural cores for computation of each neural network layer of the plurality of neural network layers;
generating microcode; and
distributing the microcode to the chip, wherein:
the chip is adapted to execute the microcode to compute the plurality of neural network layers.