Tensor processing using low precision format
Aspects of the present invention are directed to computer-implemented techniques for improving the training of artificial neural networks using a reduced precision (e.g., float16) data format. Embodiments of the present invention rescale tensor values prior to performing matrix operations (such as matrix multiplication or matrix addition) to prevent overflow and underflow. To preserve accuracy throughout the performance of the matrix operations, the scale factors are defined using a novel data format to represent tensors, wherein a matrix is represented by the tuple X, where X=(a, v[.]), wherein a is a float scale factor and v[.] are scaled values stored in the float16 format. The value of any element X[i] according to this data format would be equal to a*v[i].
1. A computer-implemented method comprising:
receiving training data for training a neural network;
applying the training data to the neural network;
accessing a first matrix, XI, and a second matrix, YI, wherein XI and YI include values represented in a reduced precision floating point format from a low threshold to a high threshold;
determining, based at least on the values, one of an overflow or an underflow would result from a matrix computation using XI and YI;
based at least in least on the determining one of the overflow or the underflow would result, determining, based at least on one or more conditions associated with avoiding the overflow or the underflow, a first scaling factor, a, for XI and a second scaling factor, b, for YI;
generating a first updated matrix, X2, by at least updating the values of X1 using a and a second updated matrix, Y2, by at least updating the values of Y1 using b;
generating, based at least on performing a matrix computation using a*X2 and b*Y2 as inputs, an output that does not result in one of the overflow or the underflow; and
updating, based at least on the output, at least one of one or more weights or one or more biases associated with the neural network.
2. The method as described in claim 1 , wherein the reduced precision floating point format comprises a float16 floating point format.
3. The computer-implemented method as described in claim 1 , wherein the matrix computation comprises at least one of: matrix multiplication; matrix convolution; or matrix addition.
4. The computer-implemented method as described in claim 1 , wherein the matrix computation is performed using a hardware computation unit of a graphics processing unit.
5. The computer-implemented method as described in claim 1 , wherein the determining the first scaling factor, a, comprises:
computing a maximum absolute value, max (X), of the values of the first matrix, XI;
computing a mean value, mean (X), of non-zero values of the values of the first matrix, XI; and
computing a range value, range (X), as [max (X)/mean (X)].
6. The computer-implemented method as described in claim 5 , wherein the determining the second scaling factor, b, comprises:
computing a maximum absolute value, max (Y), of the values of the second matrix, YI;
computing a mean value, mean (Y), of non-zero values from the values of the second matrix, YI; and
computing a range value, range (Y), as [max (Y)/mean (Y)].
7. The computer-implemented method as described in claim 6 , wherein the matrix computation comprises matrix multiplication, and wherein the determining the first scaling factor, a, and the second scaling factor, b, further comprises:
selecting the a and the b such that:
[a*mean (X) * b*mean (Y)] is equal to or greater than the low threshold;
[a*mean (X) * b*max (Y)] is equal to or less than the high threshold;
[b*mean (Y) * a*max (X)] is equal to or less than the high threshold;
K* [a*mean (X) * b*mean (Y)] is equal to or less than the high threshold, wherein K is a constant;
a*max (X) is equal to or less than the high threshold; and
b*max (Y) is equal to or less than the high threshold.
8. The computer-implemented method as described in claim 6 , wherein the matrix computation comprises matrix convolution, and wherein the determining the first scaling factor, a, and the second scaling factor, b, further comprises:
selecting the a and the b such that:
[a*mean (X) * b*mean (Y)] is equal to or greater than the low threshold;
[a*mean (X) * b*max (Y)] is equal to or less than the high threshold;
[a*max (X) * b*max (Y)] is equal to or less than the high threshold;
K* [a*mean (X) * b*mean (Y)] is equal to or less than the high threshold, wherein K is a constant;
a*max (X) is equal to or less than the high threshold; and
b*max (Y) is equal to or less than the high threshold.
9. The computer-implemented method of claim 1 , further comprising performing one or more inference operations based at least on provided input data and using the neural network with the at least one of the one or more weights or the one or more biases updated.
10. The computer-implemented method of claim 1 , further comprising determining the one or more conditions based at least on one or more of a minimum threshold or a maximum threshold that will avoid one of the overflow or the underflow.
11. A computer-implemented method comprising:
receiving training data for training a neural network;
applying the training data to the neural network;
accessing a first matrix, XI, and a second matrix, YI, wherein XI and YI include values represented in a float 16 floating point format from a low threshold to a high threshold;
determining, based at least on the values, one of an overflow or an underflow would result from a matrix computation using XI and YI;
based at least on the determining one of the overflow or the underflow would result, determining, based at least on one or more conditions associated with avoiding the overflow and the underflow, a first scaling factor, a, for XI and a second scaling factor, b, for YI;
generating a first updated matrix, X2, by at least updating the values of X1 using a and a second updated matrix, Y2, by at least updating the values of Y1 by b;
performing computations of a forward propagation layer, a convolutional layer, an inner product layer, and a backward propagation layer, wherein the convolutional layer comprises a gradient computation that comprises a matrix computation that takes a*X2 and b*Y2 as inputs and generates a gradient output that does not result in one of the overflow or the underflow; and
updating, based at least on the gradient output, at least one of one or more weights or one or more biases associate with the neural network.
12. The computer-implemented method as described in claim 11 , wherein the matrix computation is performed using a hardware computation unit of a graphics processing unit.
13. The computer-implemented method as described in claim 12 , wherein the determining the first scaling factor, a, comprises:
computing a maximum absolute value, max (X), of the values of the first matrix, X1;
computing a mean value, mean (X), of non-zero values of the values of the first matrix, XI; and
computing a range value, range (X), as [max (X)/mean (X)].
14. The computer-implemented method as described in claim 13 , wherein the determining the second scaling factor, b, comprises:
computing a maximum absolute value, max (Y), of the values of the second matrix, Y1;
computing a mean value, mean (Y), of non-zero values from the values of the second matrix, YI; and
computing a range value, range (Y), as [max (Y)/mean (Y)].
15. The computer-implemented method as described in claim 14 , wherein the matrix computation comprises matrix multiplication, and wherein the determining the first scaling factor, a, and the determining the second scaling factor, b, further comprises:
selecting the a and the b such that:
[a*mean (X) * b*mean (Y)] is equal to or greater than the low threshold;
[a*mean (X) * b*max (Y)] is equal to or less than the high threshold;
[b*mean (Y) * a*max (X)] is equal to or less than the high threshold;
K* [a*mean (X) * b*mean (Y)] is equal to or less than the high threshold, wherein K is a constant;
a*max (X) is equal to or less than the high threshold; and
b*max (Y) is equal to or less than the high threshold.
16. The computer-implemented method as described in claim 14 , wherein the matrix computation comprises matrix convolution, and wherein the determining the first scaling factor, a, and the second scaling factor, b, further comprises:
selecting the a and the b such that:
[a*mean (X) * b*mean (Y)] is equal to or greater than the low threshold;
[a*mean (X) * b*max (Y)] is equal to or less than the high threshold;
[a*max (X) * b*max (Y)] is equal to or less than the high threshold;
K* [a*mean (X) * b*mean (Y)] is equal to or less than the high threshold, wherein K is a constant;
a*max (X) is equal to or less than the high threshold; and
b*max (Y) is equal to or less than the high threshold.
17. A system comprising:
one or more processing units to:
receive training data for training a neural network;
apply the training data to the neural network;
access a first matrix, XI, and a second matrix, YI, wherein XI and YI include values represented in a float 16 floating point format from a low threshold to a high threshold;
determine, based at least on the values, one of an overflow or an underflow would result from a matrix computation using XI and YI;
based at least on the determination of one of the overflow or the underflow would result, determine, based at least on one or more conditions associated with avoiding the overflow or the underflow, a first scaling factor, a, for XI and a second scaling factor, b, for YI;
generate a first updated matrix, X2, by at least updating the values of X1 using a and a second updated matrix, Y2, by at least updating the values of Y1 using b;
perform computations of a forward propagation layer, a convolutional layer, an inner product layer, and a backward propagation layer, wherein the convolutional layer comprises a gradient computation that comprises a matrix computation that takes a*X2 and b*Y2 as inputs and generates a gradient output that does not result in one of the overflow or the underflow; and
update, based at least on the gradient output, at least one of one or more weights or one or more biases associated with the neural network.
18. The system as described in claim 17 , wherein the determination of the first scaling factor, a, comprises:
computing a maximum absolute value, max (X), of the values of the first matrix, XI;
computing a mean value, mean (X), of non-zero values of the values of the first matrix, XI; and
computing a range value, range (X), as [max (X)/mean (X)].
19. The system as described in claim 18 , wherein the determination of the second scaling factor, b, comprises:
computing a maximum absolute value, max (Y), of the values of the second matrix, Y1;
computing a mean value, mean (Y), of non-zero values from the values of the second matrix, YI; and
computing a range value, range (Y), as [max (Y)/mean (Y)].
20. The system of claim 17 , wherein the one or more processing units are further to perform one or more inference operations based at least on provided input data and using the neural network with the at least one of the one or more weights or the one or more biases updated.