Computational units for batch normalization
Herein are disclosed computation units for batch normalization. A computation unit may include a first circuit to traverse a batch of input elements x i having a first format, to produce a mean μ 1 in the first format and a mean μ 2 in a second format, the second format having more bits than the first format. The computation unit may further include a second circuit operatively coupled to the first circuit to traverse the batch of input elements x i to produce a standard deviation σ for the batch using the mean μ 1 in the first format. The computation unit may also include a third circuit operatively coupled to the second circuit to traverse the batch of input elements x i to produce a normalized set of values y i using the mean μ 2 in the second format and the standard deviation σ.
1. A reconfigurable data processor comprising:
a first circuit comprising a set of one or more computation units configured to respectively receive separate subsets of a batch of input elements x i having a first format at individual lanes of the first set of computation units and produce a mean μ 1 in the first format and a mean μ 2 in a second format, the second format having more bits than the first format;
a second circuit operatively coupled to the first circuit to traverse the batch of input elements x i to produce a standard deviation σ for the batch using the mean μ 1 in the first format; and
a third circuit operatively coupled to the second circuit to traverse the batch of input elements x i to produce a normalized set of values y i using the mean μ 2 in the second format and the standard deviation σ;
wherein at least one of the computation units of the first set of computation units comprises a multi-lane, multi-stage computation pipeline.
2. The reconfigurable data processor of claim 1 , wherein the third circuit computes differences x i −μ 2 , and multiplies the differences by a factor which is a function of a variance σ 2 in the first format.
3. The reconfigurable data processor of claim 2 , wherein the factor is an inverse of (σ 2 +ϵ), where ϵ is a machine constant that is used to avoid division by zero.
4. The reconfigurable data processor of claim 1 , wherein the third circuit computes input elements x i from the first format to the second format by computing the product of each input element x i in the first format and the identity operator in the first format.
5. The reconfigurable data processor of claim 1 , wherein the third circuit produces two normalized elements y 0 and y 1 for each input element x i .
6. The reconfigurable data processor of claim 5 , wherein the third circuit compresses the two normalized elements to produce a single output.
7. The reconfigurable data processor of claim 1 , wherein the third circuit produces a scaled and shifted normalized set of elements in the second format.
8. The reconfigurable data processor of claim 1 , wherein the third circuit performs stochastic rounding to convert the normalized set of elements to the first format.
9. The reconfigurable data processor of claim 1 , wherein the second circuit traverses the batch of input elements x i to compute for each x i :
a difference x i −μ 1 ;
a square of the difference, (x i −μ 1 ) 2 ; and
a sum of the square of the difference and a constant term ϵ, (x i −μ 1 ) 2 +ϵ.
10. The reconfigurable data processor of claim 9 , wherein the second circuit-further traverses the batch of input elements x i to compute:
an accumulated sum of the sums, (x i −μ 1 ) 2 +ϵ; and
an inverse square root of the accumulated sum.
11. The reconfigurable data processor of claim 1 , wherein the first circuit modifies the mean μ 2 in the second format to produce the mean μ 1 in the first format.
12. The reconfigurable data processor of claim 11 , wherein the first circuit modifies the mean μ 2 by truncating at least one bit from the mean μ 2 in the second format.
13. The reconfigurable data processor of claim 1 , further comprising an array of computation units that includes the first set of computation units, a second set of one or more computation units comprising the second circuit, and a third set one or more computation units comprising the third circuit.
14. The reconfigurable data processor of claim 1 , the first set of computation units comprising:
a subset of computation units configured to produce the mean μ 2 in the second format, the subset of computation units including the at least one of the computation units that comprises the multi-lane, multi-stage computation pipeline; and
another computation unit configured to produce a mean μ 1 in the first format by modifying the mean μ 2 in the second format.
15. The reconfigurable data processor of claim 1 , wherein:
the individual lanes of the first set of computation units that receive the separate subsets of the batch of input elements x i have a first stage of their multi-stage computation pipeline configured to perform Σx i /m, where m is the number of input elements xi in the batch; and
at least some later stages of the multi-stage computation pipelines of the first set of computation units are configured as a reduction tree to generate a single accumulated result for the mean μ 2 in the second format.
16. A method, comprising:
traversing, by a first circuit comprising a set of one or more computation units, a batch of input elements x i having a first format, to produce a mean μ 1 in the first format and a mean μ 2 in a second format, the second format having more bits than the first format, wherein at least one computation unit of the first set of computation units comprises a multi-lane, multi-stage computation pipeline that respectively receives a separate subset of the batch of input elements x i at individual lanes;
traversing, by a second circuit, the batch of input elements x i to produce a standard deviation σ for the batch using the mean μ 1 in the first format; and
traversing, by a third circuit, the batch of input elements x i to produce a normalized set of elements y i using the mean μ 2 in the second format and the standard deviation σ.
17. The method of claim 16 , wherein producing a normalized set of elements y i further comprises:
computing differences x i −μ 2 ; and
multiplying the differences by a factor which is a function of a variance σ 2 in the first format.
18. The method of claim 17 , wherein the factor is an inverse of (σ 2 +ϵ), where ϵ is a constant.
19. The method of claim 16 , further comprising converting input elements x i from the first format to the second format by computing the product of each input element x i in the first format and the identity operator in the first format.
20. The method of claim 16 , further comprising producing a scaled and shifted normalized set of elements in the second format.
21. The method of claim 16 , wherein the first circuit, the second circuit, and the third circuit each comprise one or more computation units in an array of computation units of a reconfigurable data processor.
22. The method of claim 16 , further comprising:
calculating, at a first stage of the multi-stage computation pipeline of the individual lanes of the first set of computation units that receive the separate subsets of the batch of input elements x i , Σx i /m, where m is the number of input elements x i in the batch; and
generating a single accumulated result for the mean μ 2 in the second format using at least some later stages of the multi-stage computation pipelines of the first set of computation units configured as a reduction tree.
23. The method of claim 22 , further comprising:
producing the mean μ 2 in the second format in a subset of the first set of computation units that includes the at least one of the computation units that comprises the multi-lane, multi-stage computation pipeline; and
producing the mean μ 1 in the first format by modifying the mean μ 2 in the second format in another computation unit of the first set of computation unit outside of the subset.