Hardware accelerator with configurable tensor operation pipeline
A hardware accelerator is disclosed that can flexibly be configured to support differing data types and differing operation flows. The hardware accelerator includes a plurality of fixed tensor operation logic units, tensor operation pipeline logic configured to receive from the processor a pipeline command including a software-defined tensor operation pipeline definition defining a plurality of tensor operation stages in a tensor operation pipeline and associated predetermined tensor operations to be performed at each of the defined tensor operation stages. The hardware accelerator is further configured to receive tensor data to be computed by the tensor operation pipeline, and implement the tensor operation pipeline to perform the tensor operations in each of the tensor operation stages on the tensor data, to thereby produce a tensor operation pipeline result for the tensor data, and output the tensor operation pipeline result to the processor.
1 . A hardware accelerator for use with a processor of a computing system, comprising:
a configurable pipeline processing element array including a plurality of processing elements, each processing element including a configurable plurality of fixed tensor operation logic units, the configurable pipeline processing element array being configured to receive a tensor operation pipeline definition and tensor data from the processor, wherein
each processing element is configured to process the tensor data by implementing a configurable tensor operation pipeline including one or more of the fixed tensor operation logic units according to the tensor operation pipeline definition;
the tensor operation pipeline definition defines a plurality of stages, each stage specifying a corresponding one of the configurable plurality of fixed tensor operation logic units;
the stages are in a predetermined order defined by an on-chip hardware layout, and individual fixed tensor operation logic units can be turned on or off by command;
at least one of the stages includes a lookup table logic unit as the fixed tensor operation logic unit for that stage; and
the configurable pipeline processing element array is configured to output a tensor operation pipeline result based on the processing of the tensor data by each tensor operation pipeline in each processing element.
2 . The hardware accelerator of claim 1 , wherein the configurable plurality of fixed tensor operation logic units are selected from the group consisting of a split logic unit, add logic unit, subtract logic unit, select logic unit, concatenate logic unit, and the lookup table logic unit.
3 . The hardware accelerator of claim 1 , wherein
the tensor data is encoded with a distribution encoding, and
the tensor operation pipeline decodes the distribution encoding.
4 . The hardware accelerator of claim 1 , wherein
the tensor operation pipeline performs block scaling on the tensor data.
5 . The hardware accelerator of claim 1 , wherein the tensor operation logic units that form the tensor operation pipeline are separate from a tensor arithmetic unit of the hardware accelerator.
6 . The hardware accelerator of claim 5 , wherein the tensor operation pipeline result is passed to the tensor arithmetic unit for further on-chip processing prior to outputting the tensor operation pipeline result.
7 . A hardware accelerator for use with a processor of a computing system, comprising:
a configurable plurality of fixed tensor operation logic units configured to perform a plurality of predetermined types of tensor operations; and
tensor operation pipeline logic configured to:
receive from the processor a pipeline command including a software-defined tensor operation pipeline definition defining a plurality of tensor operation stages in a tensor operation pipeline and associated predetermined tensor operations to be performed at each of the defined tensor operation stages, wherein:
each of the tensor operation stages specifies a corresponding one of the configurable plurality of fixed tensor operation logic units;
the tensor operation stages are in a predetermined order defined by an on-chip hardware layout, and individual fixed tensor operation logic units can be turned on or off by command; and
at least one of the tensor operation stages includes a lookup table logic unit as the fixed tensor operation logic unit for that tensor operation stage;
receive tensor data to be computed by the tensor operation pipeline, and
implement the tensor operation pipeline to perform the tensor operations in each of the tensor operation stages on the tensor data, to thereby produce a tensor operation pipeline result for the tensor data, and output the tensor operation pipeline result to the processor.
8 . The hardware accelerator of claim 7 , wherein the tensor data includes numerical parameters of a neural network.
9 . The hardware accelerator of claim 8 , wherein the numerical parameters of the neural network are floating point values including one or more mantissa bits and one or more exponent bits.
10 . The hardware accelerator of claim 7 , wherein the predetermined types of tensor operations are selected from the group consisting of split, add, subtract, select, concatenate, and perform a lookup to a lookup table.
11 . The hardware accelerator of claim 10 , wherein the lookup table is programmable to implement a user-defined function.
12 . The hardware accelerator of claim 11 , wherein the user-defined function is block scaling or decoding a distribution encoding of the tensor data.
13 . The hardware accelerator of claim 10 , wherein the tensor data includes floating point values and the split function splits floating point values into constituent mantissa and exponent portions.
14 . A computing system comprising:
a processor, and
a hardware accelerator communicatively coupled to the processor, the hardware accelerator including a configurable plurality of fixed tensor operation units configured to perform a plurality of predetermined types of tensor operations, wherein
the hardware accelerator is configured to receive from the processor a pipeline command including a software-defined tensor operation pipeline definition defining a plurality of tensor operation stages in a tensor operation pipeline and associated predetermined tensor operations to be performed at each of the defined tensor operation stages,
each of the tensor operation stages specifies a corresponding one of the configurable plurality of fixed tensor operation logic units,
the tensor operation stages are in a predetermined order defined by an on-chip hardware layout, and individual fixed tensor operation logic units can be turned on or off by command,
at least one of the tensor operation stages includes a lookup table logic unit as the fixed tensor operation logic unit for that tensor operation stage,
the hardware accelerator is further configured to receive tensor data to be computed by the tensor operation pipeline, and
in response to receiving the tensor pipeline command and the tensor data, the hardware accelerator is configured to implement the tensor operation pipeline to perform the tensor operations in each of the tensor operation stages on the tensor data, to thereby produce a tensor operation pipeline result for the tensor data, and output the tensor operation pipeline result to the processor.
15 . The computing system of claim 14 , wherein the tensor data includes numerical parameters of a neural network.
16 . The computing system of claim 15 , wherein the numerical parameters of the neural network are floating point values including one or more mantissa bits and one or more exponent bits.
17 . The computing system of claim 14 , wherein the tensor operations are selected from the group consisting of split, add, subtract, select, concatenate, and perform a lookup to a lookup table.
18 . The computing system of claim 17 , wherein the tensor operations include the lookup table, and the lookup table is programmable to implement a user-defined function.
19 . The computing system of claim 17 , wherein the tensor data includes floating point values and the split function splits floating point values into constituent mantissa and exponent portions.