Systolic array including fused multiply accumulate with efficient prenormalization and extended dynamic range
Systems and methods are provided to perform multiply-accumulate operations of at least one normalized number in a systolic array. The systolic array can obtain a first input and detect that the first input is denormal. Based on determining the first input is denormal, the systolic array can generate a first normalized number by normalizing the first input. Processing elements of the systolic array can include a multiplier and an adder. The multiplier can multiply the first normalized number by a second normal or normalized number to generate a multiplier product and the adder can add an input partial sum to the multiplier product to generate an addition result.
1 . A systolic circuit comprising:
a systolic array comprising processing elements; and
a normalizer configured to:
receive a first input represented in floating-point with a first bit-length;
receive a second input represented in floating-point with a second bit-length, wherein the second bit-length is greater than the first bit-length; and
generate a first normalized number represented in floating-point with the second bit-length;
wherein an individual processing element of the systolic array is configured to:
perform one or more operations on the first normalized number to generate an output.
2 . The systolic circuit of claim 1 , wherein the first input and the second input are normal, and wherein to generate the first normalized number, the normalizer is configured to:
generate the first normalized number based on the first input.
3 . The systolic circuit of claim 1 , wherein the first input is denormal, and wherein to generate the first normalized number, the normalizer is configured to:
generate the first normalized number based on the first input.
4 . The systolic circuit of claim 1 , wherein to generate the first normalized number, the normalizer is configured to:
expand a numerical range of an exponent of the first input.
5 . The systolic circuit of claim 1 , wherein the normalizer is configured to:
detect by a denormal detector of the normalizer, that the first input is denormal, and
wherein to generate the first normalized number, the normalizer is configured to:
generate the first normalized number based on the first input.
6 . The systolic circuit of claim 1 , wherein the second input comprises a second number, and wherein to perform the one or more operations on the first normalized number, the individual processing element is configured to:
multiply the first normalized number by the second number to generate a multiplier product; and
add an input partial sum with the multiplier product to generate an addition result.
7 . The systolic circuit of claim 1 , wherein the normalizer is configured to:
generate a second normalized number represented in floating-point with the second bit-length, wherein the second normalized number comprises the second input.
8 . The systolic circuit of claim 1 , wherein the normalizer comprises a first normalizer and a second normalizer.
9 . The systolic circuit of claim 1 , wherein the normalizer is configured to support at least n-bit floating-point numbers, wherein n can be any number, and wherein individual processing elements of the systolic array comprise:
a multiplier configured to multiply at least two n-bit numbers; and
an adder configured to add two m-bit numbers, wherein m is greater than n.
10 . The systolic circuit of claim 1 , wherein the individual processing element comprises:
a multiplier configured to multiply at least two numbers having the second bit-length; and
an adder configured to add two numbers having a third bit-length, wherein the third bit-length is greater than the second bit-length.
11 . The systolic circuit of claim 1 , wherein the processing elements are arranged into a plurality of rows.
12 . The systolic circuit of claim 1 , wherein the normalizer comprises at least one of:
a leading zero encoder or counter configured to detect a number of leading zeros in a significand of the first input;
a exponent expander configured to expand a numerical range of an exponent of the first input; or
a shifter configured to shift a significand of the first input based on the number of leading zeros in the significand of the first input.
13 . The systolic circuit of claim 1 , wherein, the individual processing element is configured to:
route the output to a subsequent processing element of the processing elements as an input.
14 . The systolic circuit of claim 1 , wherein the systolic array performs neural network computations.
15 . A method for systolic processing by a processing element in a systolic array of processing elements, the method comprising:
obtaining, by a normalizer of the processing element, a first input represented in floating-point with a first bit-length;
obtaining, by the normalizer, a second input represented in floating-point with a second bit-length;
generating, by the normalizer, a first normal number represented in floating-point with the first bit-length; and
performing, by the processing element, one or more operations on the first normal number to generate an output.
16 . The method of claim 15 , wherein generating the first normal number comprises:
generating, using an exponent expander of the normalizer, the first normal number.
17 . The method of claim 15 , wherein generating the first normal number comprises:
generating, by a first normalizer of the normalizer, the first normal number based on the first input,
the method further comprising one of:
generating, by a second normalizer of the normalizer, a second normal number represented in floating-point with the second bit-length, wherein the second normal number comprises the second input; or
generating, by the first normalizer, a second normal number represented in floating-point with the second bit-length, wherein the second normal number comprises the second input.
18 . Non-transitory computer-readable media including computer-executable instructions that, when executed by a systolic circuit comprising a normalizer and an array of processing elements, cause the systolic circuit to:
obtain, by the normalizer, a first input represented in floating-point with a first bit-length;
obtain, by the normalizer, a second input represented in floating-point with a second bit-length;
generate, by the normalizer, a first normal number represented in floating-point with the first bit-length; and
perform, by a processing element of the array of processing elements, one or more operations on the first normal number to generate an output.
19 . The non-transitory computer-readable media of claim 18 , wherein the normalizer comprises a denormal detector.
20 . The non-transitory computer-readable media of claim 18 , wherein individual processing elements of the array of processing elements further comprise:
a multiplier; and
an adder,
wherein the multiplier and the adder lack support for inputs provided in denormal form.