Special purpose neural network training chip
Methods, systems, and apparatus including a special purpose hardware chip for training neural networks are described. The special-purpose hardware chip may include a scalar processor configured to control computational operation of the special-purpose hardware chip. The chip may also include a vector processor configured to have a 2-dimensional array of vector processing units which all execute the same instruction in a single instruction, multiple-data manner and communicate with each other through load and store instructions of the vector processor. The chip may additionally include a matrix multiply unit that is coupled to the vector processor configured to multiply at least one two-dimensional matrix with a second one-dimensional vector or two-dimensional matrix in order to obtain a multiplication result.
1. A special-purpose hardware chip for training neural networks, the special-purpose hardware chip comprising:
a scalar processor configured to control computational operation of the special-purpose hardware chip;
a vector processor configured to have a 2-dimensional array of vector processing units which all execute the same instruction in a single instruction, multiple-data manner and communicate with each other through load and store instructions of the vector processor;
a matrix multiply unit that is coupled to the vector processor configured to multiply at least one two-dimensional matrix with a second one-dimensional vector or two-dimensional matrix in order to obtain a multiplication result, and
a reduction and permutation unit configured to perform a reduction on numbers and permute the numbers among different lanes of the 2-dimensional array.
2. The special-purpose hardware chip of claim 1 , further comprising:
a vector memory configured to provide private memory to the vector processor.
3. The special-purpose hardware chip of claim 1 , further comprising:
a scalar memory configured to provide private memory to the scalar processor.
4. The special-purpose hardware chip of claim 1 , further comprising:
a transpose unit configured to perform a transposition operation of a matrix.
5. The special-purpose hardware chip of claim 1 , further comprising:
a high-bandwidth memory configured to store data of the special-purpose hardware chip.
6. The special-purpose hardware chip of claim 1 , further comprising a sparse computation core.
7. The special-purpose hardware chip of claim 1 , further comprising:
an interface; and
an inter-chip interconnect, which connects the interface or resources on the special-purpose hardware chip to other special-purpose hardware chips or resources.
8. The special-purpose hardware chip of claim 7 , further comprising:
a plurality of high bandwidth memory; wherein the inter-chip interconnect connects the interface and one or more of the plurality of high bandwidth memory to other special-purpose hardware chips.
9. The special-purpose hardware chip of claim 7 , wherein the interface is a host interface to a host computer.
10. The special-purpose hardware chip of claim 7 , wherein the interface is a standard network interface to a network of host computers.
11. The special-purpose hardware chip of claim 7 , comprising:
a scalar memory, a vector memory, said scalar processor, said vector processor, and said matrix multiply unit, wherein said scalar processor performs very long instruction word (VLIW) instruction fetch/execute loop and controls said special-purpose hardware chip, wherein after fetching and decoding an instruction bundle, said scalar processor itself only executes instructions found in scalar slots of the instruction bundle using multiple, multi-bit registers of the scalar processor and scalar memory, wherein a scalar instruction set of the instructions found in the scalar slots includes arithmetic operations used in address calculations, load/store instructions, and branch instructions, and wherein the remaining instruction slots encode instructions for the vector processor and said matrix multiply unit.