Methods and apparatus to perform tensor operations
A network of matrix processing units (MPUs) is provided on a device, where each MPU is connected to at least one other MPU in the network, and each MPU is to perform matrix multiplication operations. Computer memory stores tensor data and a master control central processing unit (MCC) is provided on the device to receive an instruction from a host device, where the instruction includes one or more tensor operands based on the tensor data. The MCC invokes a set of operations on one or more of the MPUs based on the instruction, where the set of operations includes operations on the tensor operands. A result is generated from the set of operations, the result embodied as a tensor value.
1 . An apparatus comprising:
interface circuitry to communicate with a network of processing units, wherein each processing unit is connected to at least one other processing unit in the network, and each processing unit is to perform matrix multiplication operations;
a memory comprising high-bandwidth memory (HBM) to store tensor data; and
processor circuitry to:
obtain an instruction received from a host processor coupled to the apparatus, wherein the instruction indicates one or more tensor operands corresponding to the tensor data;
partition the tensor data into a number of partitions based on a number of processing units in the network of processing units;
cause the interface circuitry to distribute the partitioned tensor data from the HBM to memory resources blocks of corresponding processing units in the network of processing units to perform partial matrix multiplication operations on the partitioned tensor data and to output partial tensor operands of the instruction from a processing unit to neighboring processing units of the processing unit in the network of processing units;
invoke a set of operations including pre-arithmetic operations that comprise an addition operation on the tensor data prior to the partial matrix multiplication operations, post-arithmetic operations after the partial matrix multiplication operations, and a matrix-wide operation on the processing units based on the instruction, wherein the set of operations includes operations on the tensor operands, and the matrix-wide operation processes data among elements of a single matrix; and
output a result of the set of operations, wherein the result includes a tensor value to be stored in a location in a memory resource block identified by the instruction.
2 . The apparatus of claim 1 , wherein the processor circuitry is further to cause the memory to store the result, wherein the result is stored as a tensor value in the memory.
3 . The apparatus of claim 1 , wherein the processor circuitry sends the result to a host device, and the host device includes the host processor connected to the processor circuitry.
4 . The apparatus of claim 1 , wherein the processor circuitry is to select a subset of the network of processing units to perform the set of operations.
5 . The apparatus of claim 4 , wherein the subset of processing units includes two or more of the processing units.
6 . The apparatus of claim 1 , wherein the instruction includes a stream of instructions and the processor circuitry is to coordinate data flow and a sequence of operations to be performed by the network of processing units based on the stream of operations.
7 . The apparatus of claim 6 , wherein the sequence of operations includes a sequence of tensor arithmetic operations.
8 . The apparatus of claim 7 , wherein the sequence of tensor operations includes matrix-matrix operations.
9 . The apparatus of claim 1 , wherein the memory includes a memory resource block to be shared by two or more processing units in the network of processing units.
10 . The apparatus of claim 9 , wherein invoking the set of operations includes pointing one or more of the processing units to the memory resource block to access the tensor data.
11 . The apparatus of claim 10 , wherein the set of operations include at least one of a row/column broadcast, block shifting, matrix copy, matrix transpose, and matrix expansion.
12 . The apparatus of claim 9 , wherein the memory includes a memory block to group a plurality of memory resource blocks, and two or more processing units in the network of processing units have read/write access to the plurality of memory resource blocks in the memory block.
13 . The apparatus of claim 1 , further including an on-chip router to route data multi-directionally between components of the apparatus.
14 . The apparatus of claim 1 , wherein the memory includes one or more barrel shifters to shift a matrix described in the memory to target a read or write to a particular row or column of the matrix.
15 . The apparatus of claim 1 , wherein the set of operations includes a max pooling operation.
16 . The apparatus of claim 1 , wherein the set of operations includes performing a Winograd transformation on the operands and performing a matrix multiplication on the operands transformed by the Winograd transformation.
17 . The apparatus of claim 1 , wherein each the tensor operands includes a matrix.
18 . The apparatus of claim 1 , wherein the tensor operands include a particular input matrix and the set of operations includes a matrix dimension shuffle operation to reorder a plurality of dimensions of the particular input matrix.
19 . The apparatus of claim 1 , wherein at least a particular processing unit in the network of processing units includes local memory to store a set of matrix subroutines, and the particular processing unit is to:
translate an operation received from the processor circuitry into a subset of the matrix subroutines; and
perform the operation through execution of the subset of the matrix subroutines.
20 . The apparatus of claim 1 , wherein the set of operations are used to implement one of a set of deep learning models, and the set of deep learning models includes a multilayer perceptron model, a restricted Boltzmann machine model, a deep belief network model, an auto-encoder model, and a convolutional neural network.
21 . A method comprising:
storing tensor data in memory comprising high-bandwidth memory (HBM), wherein the memory is accessible to a network of processing units of an apparatus;
obtaining an instruction from a host processor coupled to the apparatus, wherein the instruction indicates one or more tensor operands corresponding to the tensor data;
partitioning the tensor data into a number of partitions based on a number of processing units in the network of processing units;
distributing the partitioned tensor data from the HBM to memory resource blocks of corresponding processing units in the network of processing units;
causing the processing units to perform partial matrix multiplication operations on the partitioned tensor data and to output partial tensor operands of the instruction from a processing unit to neighboring processing units of the processing unit in the network of processing units; and
generating a result from performance of the partial matrix multiplication operations, the operations including pre-arithmetic operations that comprise an addition operation on the tensor data prior to the partial matrix multiplication operations, post-arithmetic operations after the partial matrix multiplication operations, and a matrix-wide operation, wherein the result includes a tensor value to be stored in a location in a memory resource block identified by the instruction, and the matrix-wide operation processes data among elements of a single matrix.
22 . A system comprising:
a port to connect to a host processor;
a plurality of processing units, wherein each processing unit includes circuitry to perform tensor arithmetic operations;
a memory comprising high-bandwidth memory (HBM) to store tensor data; and
processor circuitry to:
obtain an instruction from the host processor, the instruction indicating one or more tensor operands corresponding to the tensor data;
partition the tensor data into a number of partitions based on a number of processing units in the plurality of processing units;
distribute the partitioned tensor data from the HBM to memory resource blocks of corresponding processing units in the plurality of processing units;
cause the processing units to perform partial matrix multiplication operations on the partitioned tensor data and to output partial tensor operands of the instruction from a processing unit to neighboring processing units of the processing unit in the plurality of processing units;
cause the one or more of the processing units to perform a set of operations including pre-arithmetic operations that comprise an addition operation on the tensor data prior to the partial matrix multiplication operations, post-arithmetic operations after the partial matrix multiplication operations, and a matrix-wide operation based on the instruction, wherein the set of operations include operations on the tensor operands, and the matrix-wide operation processes data among elements of a single matrix; and
return a result of the set of operations to the host processor, wherein the result includes a tensor value to be stored in a location in a memory resource block identified by the instruction.
23 . The system of claim 22 , further including the host processor.
24 . The system of claim 23 , wherein the system is implemented using a system on a chip.
25 . The system of claim 23 , wherein the system is implemented using a server blade.