Hierarchical parallelism in a network of distributed neural network cores
Networks of distributed neural cores are provided with hierarchical parallelism. In various embodiments, a plurality of neural cores is provided. Each of the plurality of neural cores comprises a plurality of vector compute units configured to operate in parallel. Each of the plurality of neural cores is configured to compute in parallel output activations by applying its plurality of vector compute units to input activations. Each of the plurality of neural cores is assigned a subset of output activations of a layer of a neural network for computation. Upon receipt of a subset of input activations of the layer of the neural network, each of the plurality of neural cores computes a partial sum for each of its assigned output activations, and computes its assigned output activations from at least the computed partial sums.
1 . A neural inference chip, comprising:
a plurality of neural cores comprising a first neural core and a second neural core, each neural core of the plurality of neural cores comprising a plurality of vector compute units and a parallel sum and nonlinearity unit such that the first neural core comprises a first plurality of vector compute units and a first parallel sum and nonlinearity unit and the second plurality of vector compute units comprises a second plurality of vector compute units and a second parallel sum and nonlinearity unit, wherein:
each of the plurality of neural cores is configured to compute in parallel output activations by applying its plurality of vector compute units to input activations and applying a nonlinear function by its parallel sum and nonlinearity unit such that the first neural core is configured to compute output activations by applying the first plurality of vector compute units to input activations and applying a nonlinear function by the first parallel sum and nonlinearity unit;
each output activation of a layer of an artificial neural network is assigned to at least one neural core of the plurality of neural cores such that the first neural core is assigned to a subset of output activations of the layer for computation;
the first and second pluralities of vector compute units each comprise multiplication and addition units;
the second neural core of the plurality of neural cores computes zero and non-zero partial sums;
the second neural core directly communicates its non-zero partial sums with the first parallel sum and nonlinearity unit of via a wire without communicating its zero partial sums.
2 . The neural inference chip of claim 1 , wherein:
upon receipt of a subset of input activations of the layer of the artificial neural network, each of the plurality of neural cores
computes a partial sum for each of its assigned output activations, and
computes its assigned output activations from at least the computed partial sums.
3 . The neural inference chip of claim 2 , wherein upon receipt of a subset of input activations of the layer of the artificial neural network, each of the plurality of neural cores
receives partial sums for at least one of its assigned output activations from another of the plurality of neural cores, and
computes its assigned output activations from the computed partial sums and the received partial sums.
4 . The neural inference chip of claim 2 , wherein the plurality of neural cores perform said partial sum computation in parallel.
5 . The neural inference chip of claim 2 , wherein the plurality of neural cores perform said output activation computation in parallel.
6 . The neural inference chip of claim 2 , wherein computing the partial sum comprises applying at least one of the plurality of vector compute units to multiply the input activations and synaptic weights.
7 . The neural inference chip of claim 2 , wherein computing the assigned output activations comprises applying a plurality of addition units.
8 . The neural inference chip of claim 2 , wherein the plurality of vector compute units are configured to:
perform a plurality of multiply operations in parallel;
perform a plurality of additions in parallel; and
accumulate the partial sum.
9 . The neural inference chip of claim 2 , wherein the plurality of vector compute units are configured to compute partial sums in parallel.
10 . The neural inference chip of claim 1 , wherein the plurality of vector compute units comprise accumulation units.
11 . The neural inference chip of claim 1 , wherein said computation of the subset of output activations of a layer of the artificial neural network by each of the plurality of neural cores is pipelined.
12 . The neural inference chip of claim 11 , wherein each of the plurality of neural cores is configured to concurrently perform each stage of a plurality of stages of said computation of the subset of output activations of a layer of the artificial neural network.
13 . The neural inference chip of claim 12 , wherein said computation of the subset of output activations of a layer of the artificial neural network maintains parallelism.
14 . The neural inference chip of claim 1 , wherein each neural core of the plurality of neural cores computes its zero and non-zero partial sums simultaneously and separately computes all of its assigned subset of output activations simultaneously.
15 . A method comprising:
receiving, at each neural core of a plurality of neural cores, a subset of input activations of a layer of an artificial neural network such that a first subset of input activations of a first layer is received at a first neural core of the plurality, wherein:
each neural core of the plurality of neural cores comprises a plurality of vector compute units and a parallel sum and nonlinearity unit such that the first neural core comprises a first plurality of vector compute units and a first parallel sum and nonlinearity unit and the second plurality of vector compute units comprises a second plurality of vector compute units and a second parallel sum and nonlinearity unit, and
each of the plurality of neural cores is configured to compute in parallel output activations by applying its plurality of vector multipliers to its subset of input activations and applying a nonlinear function by its parallel sum and nonlinearity unit such that the first neural core is configured to compute output activations by applying the first plurality of vector compute units to the first subset of input activations of the first layer and applying a nonlinear function by the first parallel and nonlinearity unit;
assigning each output activation of the first layer to a neural core of the plurality of neural cores such that the first neural core is assigned to a first subset of output activations of the first layer for computation; and
upon receipt of the first subset of input activations of the layer of the artificial neural network, the first neural core
computing a partial sum for each output activation of the first subset of output activations, and
computing the first subset of output activations from at least the computed partial sums, wherein
the first and second plurality of vector compute units each comprise multiplication and addition units,
the second neural core of the plurality of neural cores computes zero and non-zero partial sums, and
the second neural core directly communicates its non-zero partial sums with the first parallel sum and nonlinearity unit of via a wire without communicating its zero partial sums.
16 . The method of claim 15 , wherein upon receipt of a subset of input activations of the layer of the artificial neural network, each of the plurality of neural cores
receives non-zero partial sums for at least one of its assigned output activations from another of the plurality of neural cores, and
computes its assigned output activations from the computed partial sums and the received non-zero partial sums.
17 . The method of claim 15 , wherein the plurality of vector compute units comprise accumulation units.
18 . The method of claim 15 , wherein the plurality of neural cores perform said partial sum computation in parallel.
19 . The method of claim 15 , wherein the plurality of neural cores perform said output activation computation in parallel.
20 . The method of claim 15 , wherein computing the partial sum comprises applying at least one of the plurality of vector compute units to multiply the input activations and synaptic weights.
21 . The method of claim 15 , wherein computing the assigned output activations comprises applying a plurality of addition units.
22 . The method of claim 15 , further comprising:
performing a plurality of multiply operations in parallel;
performing a plurality of additions in parallel; and
accumulating the partial sum.