Central scheduler and instruction dispatcher for a neural inference processor
Neural inference processors are provided. Each processor includes a plurality of cores. Each core includes a neural computation unit, an activation memory, and a local controller. The neural computation unit is adapted to apply a plurality of synaptic weights to a plurality of input activations to produce a plurality of output activations. The activation memory is adapted to store the input activations and the output activations. The local controller is adapted to load the input activations from the activation memory to the neural computation unit and to store the plurality of output activations from the neural computation unit to the activation memory. The processor includes a neural network model memory adapted to store network parameters, including the plurality of synaptic weights. The processor includes a global scheduler operatively coupled to the plurality of cores, adapted to provide the synaptic weights from the neural network model memory to each core.
1 . A system comprising:
a plurality of cores, the cores individually comprising:
a neural computation unit, the neural computation unit adapted to apply a plurality of synaptic weights to a plurality of input activations to produce a plurality of output activations,
an activation memory adapted to store the input activations and the output activations, and
a local controller, the local controller being adapted to load the input activations from the activation memory to the neural computation unit and to store the plurality of output activations from the neural computation unit to the activation memory;
a neural network model memory adapted to store the plurality of synaptic weights;
a plurality of network-on-chips (NoCs) configured to distribute the synaptic weights and instructions to the plurality of cores;
a global scheduler operatively coupled to the plurality of cores, the global scheduler adapted to:
load the synaptic weights and instructions from the neural network model memory to the cores without use of the local controller, and
communicate with the local controllers to share in scheduling operations of the system; and
a network,
wherein the network comprises a plurality of rows, respective ones of the rows being connected to a corresponding subset of the plurality of cores,
wherein each row comprises an interconnect operable to select between a broadcast input and a row-specific input,
wherein each of the cores have different associated activation data.
2 . The system of claim 1 , wherein the global scheduler is adapted to provide local controller instructions to the cores.
3 . The system of claim 1 , wherein the global scheduler provides the same synaptic weights to each of the cores.
4 . The system of claim 2 , wherein the global scheduler provides the same local controller instructions to each of the cores.
5 . The system of claim 1 , wherein the global scheduler provides different synaptic weights to at least some of the cores.
6 . The system of claim 2 , wherein the global scheduler provides different local controller instructions to at least some of the cores.
7 . The system of claim 1 , wherein the global scheduler provides the synaptic weights via the network.
8 . The system of claim 1 , wherein the global scheduler provides local controller instructions to the cores via the network.
9 . The system of claim 1 , wherein the network comprises a tree network.
10 . The system of claim 1 , wherein the network comprises a row broadcast network.
11 . The system of claim 10 , wherein the global scheduler directly provides the instructions to the neural computation units of the cores via a first of the NoCs, wherein the global scheduler directly provides the synaptic weights to the neural computation units of the cores via a second of the NoCs that is different than the first NoC.
12 . The system of claim 11 , wherein the activation data is stored as the input activations within the activation memory, wherein the synaptic weights provided via the second NoC are the same within each row of cores, wherein the synaptic weights of different rows of cores are different.
13 . The system of claim 12 wherein the interconnect is operable by the global scheduler, wherein the instructions provided via the first NoC are the same within each row of cores, wherein the instructions of different rows of cores are different.
14 . The system of claim 1 , wherein the global scheduler is adapted to provide the synaptic weights from a plurality of non-consecutive regions of the neural network model memory.
15 . A system comprising:
a neural computation unit, the neural computation unit adapted to apply a plurality of synaptic weights to a plurality of input activations to produce a plurality of output activations;
a neural network model memory adapted to store the plurality of synaptic weights;
an activation memory adapted to store the input activations and the output activations;
a local controller operatively coupled to the neural computation unit, the neural network model memory, and the activation memory; the local controller adapted to:
load the input activations from the activation memory to the neural computation unit, and store the plurality of output activations from the neural computation unit to the activation memory;
a plurality of network-on-chips (NoCs) configured to distribute the synaptic weights and instructions to the neural computation unit;
a plurality of schedulers operatively coupled to the neural computation unit, the neural network model memory, and the activation memory, the plurality of schedulers adapted to, in parallel:
load the synaptic weights and instructions from the neural network model memory to the neural computation unit without use of the local controller, and communicate with the local controller to share in scheduling operations of the system; and
a network,
wherein the network comprises a plurality of row networks, respective ones of the row networks connecting different corresponding subsets of a plurality of cores,
wherein the respective cores have different associated activation data stored as the input activations within the activation memory.
16 . A method comprising:
loading, by a global scheduler, synaptic weights and instructions from a neural network model memory to a plurality of cores without use of a local controller;
loading, by the local controllers of the cores, input activations from an activation memory of the respective core to a neural computation unit of the core;
applying, by the neural computation units of the cores, the synaptic weights to the input activations to produce a plurality of output activations; and
storing, by the local controllers of the cores, the plurality of output activations from the neural computation unit of the respective core to the activation memory of the core,
wherein the global scheduler communicates with the local controllers of the cores to share in scheduling core operations, and
wherein the global scheduler directly provides the instructions to the neural computation units of the cores via a plurality of network-on-chips (NoCs), wherein a first of the NoCs is configured to distribute the synaptic weights to the plurality of cores,
wherein a second of the NoCs that is different than the first NoC is configured to distribute the instructions to the plurality of cores,
wherein the global scheduler provides the synaptic weights via a row broadcast network,
wherein the row broadcast network is coupled to the plurality of cores,
wherein the row broadcast network comprises a plurality of rows, respective ones of the rows being connected to a corresponding unique subset of the plurality of cores,
wherein each row comprises an interconnect operable to select between a broadcast input and a row-specific input such that the respective cores have different associated activation data stored as the input activations within the activation memory,
wherein the synaptic weights provided via the second NoC are the same within each row of cores,
wherein the synaptic weights of different rows of cores are different.
17 . The method of claim 16 , further comprising:
providing, by the global scheduler, local controller instructions to the plurality of cores.
18 . The method of claim 17 , wherein the global scheduler provides the same local controller instructions to the plurality of cores.
19 . The method of claim 17 , wherein the global scheduler provides different local controller instructions to at least some of the cores.
20 . The method of claim 16 , wherein the global scheduler provides local controller instructions to cores via the row broadcast network.
21 . The method of claim 16 , wherein the global scheduler provides the synaptic weights from a plurality of non-consecutive regions of the neural network model memory.
22 . The system of claim 15 , wherein the synaptic weights and instructions are the same within the cores of the same subset of the plurality of cores, wherein the synaptic weights and instructions are different between the different subsets of the plurality of cores.