Massively parallel neural inference computing elements
Massively parallel neural inference computing elements are provided. A plurality of multipliers is arranged in a plurality of equal-sized groups. Each of the plurality of multipliers is adapted to, in parallel, apply a weight to an input activation to generate an output. A plurality of adders is operatively coupled to one of the groups of multipliers. Each of the plurality of adders is adapted to, in parallel, add the outputs of the multipliers within its associated group to generate a partial sum. A plurality of function blocks is operatively coupled to one of the plurality of adders. Each of the plurality of function blocks is adapted to, in parallel, apply a function to the partial sum of its associated adder to generate an output value.
1. A system comprising:
a plurality of multipliers, the plurality of multipliers arranged in a plurality of equal-sized groups, each of the plurality of multipliers being adapted to, in parallel, apply a weight to an input activation to generate an output;
a plurality of adders, each of the plurality of adders being operatively coupled to one of the groups of multipliers, each of the plurality of adders being adapted to, in parallel, add the outputs of the multipliers within its associated group to generate a partial sum;
a first plurality of function blocks, each of the first plurality of function blocks being operatively coupled to one of the plurality of adders, each of the first plurality of function blocks being adapted to, in parallel, apply a function to the partial sum of its associated adder to generate an output value;
a vector register, the vector register being operatively coupled to the first plurality of function blocks, the vector register adapted to store the output values of the first plurality of function blocks;
a second plurality of function blocks, each of the second plurality of function blocks being operatively coupled to the vector register, each of the second plurality of function blocks being adapted to, in parallel, apply a function to the stored output values.
2. The system of claim 1 , adapted to receive a matrix of weights and a vector of activations.
3. The system of claim 1 , wherein each of the plurality of adders comprises a tree of adders.
4. The system of claim 3 , wherein the tree of adders is a binary tree.
5. The system of claim 3 , wherein the tree of adders comprises a plurality of carry-save adders.
6. The system of claim 2 , wherein each activation of the vector of activations is broadcast to all of the groups of multipliers.
7. The system of claim 2 , further comprising a systolic pipeline operatively coupled to each of the groups of multipliers.
8. The system of claim 2 , wherein the matrix of weights is compressed, and wherein the system is adapted to decompress the compressed matrix of weights.
9. The system of claim 1 , wherein the groups of multipliers are pipelined.
10. The system of claim 1 , wherein the weights are balanced ternary values.
11. The system of claim 1 , wherein each of the plurality of multipliers comprises a multiplexor.
12. The system of claim 1 , wherein the first plurality of function blocks are adapted to combine the output values with subsequently computed output values.
13. The system of claim 1 , further comprising:
a plurality of shifters, each shifter operatively connected to one of the first plurality of function blocks, each shifter adapted to, in parallel, shift the output value of its corresponding function block, and wherein
the first plurality of function blocks are adapted to combine the shifted values with subsequently computed output values.
14. The system of claim 1 , wherein the function of each of the first plurality of function blocks is an activation function.
15. The system of claim 1 , wherein the function of each of the first plurality of function blocks is programmable.
16. The system of claim 1 , wherein the function of each of the first plurality of function blocks is addition.
17. The system of claim 1 , wherein the function of each of the first plurality of function blocks is multiplication.
18. The system of claim 1 , wherein the function of each of the first plurality of function blocks is an identity function.
19. The system of claim 1 , further comprising a lookup table, the function of each of the first plurality of function blocks comprising a lookup from the lookup table.
20. The system of claim 19 , wherein the lookup table is programmable.
21. The system of claim 1 , wherein the function of each of the first plurality of function blocks is a max function.
22. The system of claim 1 , wherein the function of each of the first plurality of function blocks is a min function.
23. The system of claim 1 , wherein the function of each of the second plurality of function blocks is the identity function.
24. The system of claim 1 , wherein the function of each of the second plurality of function blocks is an activation function.
25. The system of claim 1 , adapted to perform a permute operation.
26. The system of claim 1 , adapted to perform a vector packing operation.
27. The system of claim 1 , adapted to perform a max-pooling operation.
28. The system of claim 1 , adapted to perform an average-pooling operation.
29. The system of claim 1 , adapted to perform an element-wise multiply operation.
30. A method comprising:
applying by a plurality of equal-sized groups of multipliers, in parallel, a plurality of weights to a plurality of input activations to generate a plurality of outputs for each group of multipliers;
adding, in parallel, the plurality of outputs from each group of multipliers to generate a partial sum from each group of multipliers;
applying, in parallel, a function to the partial sums to generate a plurality of output values;
storing the plurality of output values; and
applying, in parallel, a function to the stored output values.
31. The method of claim 30 , further comprising:
shifting, in parallel, the output value of each of the first plurality function blocks; and
combining the shifted values with subsequently computed output values.
32. A system comprising:
a plurality of multipliers, the plurality of multipliers arranged in a plurality of equal-sized groups;
a plurality of adders, each of the plurality of adders being operatively coupled to one of the groups of multipliers;
a first plurality of function blocks, each of the first plurality of function blocks being operatively coupled to one of the plurality of adders;
a vector register, the vector register being operatively coupled to the first plurality of function blocks;
a second plurality of function blocks, each of the second plurality of function blocks being operatively coupled to the vector register; and
a computer readable storage medium having program instructions embodied therewith, the program instructions executable to perform a method comprising:
by each of the plurality of multipliers, in parallel, applying a weight to an input activation to generate an output;
by each of the plurality of adders, in parallel, adding the outputs of the multipliers within its associated group to generate a partial sum;
by each of the plurality of function blocks, in parallel, applying a function to the partial sum of its associated adder to generate an output value;
by the vector register, storing the output values of the first plurality of function blocks; and
by each of the second plurality of function blocks, in parallel, applying a function to the stored output values.
33. The system of claim 32 , further comprising:
a plurality of shifters, each shifter operatively connected to one of the first plurality of function blocks, wherein the method further comprises:
by each shifter, in parallel, shifting the output value of its corresponding function block;
by the first plurality of function blocks, combining the shifted values with subsequently computed output values.