Artificial neural network training using flexible floating point tensors
Thus, the present disclosure is directed to systems and methods for training neural networks using a tensor that includes a plurality of FP16 values and a plurality of bits that define an exponent shared by some or all of the FP16 values included in the tensor. The FP16 values may include IEEE 754 format 16-bit floating point values and the tensor may include a plurality of bits defining the shared exponent. The tensor may include a shared exponent and FP16 values that include a variable bit-length mantissa and a variable bit-length exponent that may be dynamically set by processor circuitry. The tensor may include a shared exponent and FP16 values that include a variable bit-length mantissa; a variable bit-length exponent that may be dynamically set by processor circuitry; and a shared exponent switch set by the processor circuitry to selectively combine the FP16 value exponent with the shared exponent.
1. A system for training a neural network, comprising:
processor circuitry;
a communications interface coupled to the processor circuitry, the communications interface couplable to the neural network; and
a storage device coupled to the processor circuitry, the storage device including machine readable instructions that, when executed by the processor circuitry, cause the processor circuitry to:
generate a neural network training tensor that includes:
a five-bit shared exponent stored in a memory, the five-bit shared exponent common to each of a plurality of 16-bit floating point values included in the neural network training tensor, each of the plurality of 16-bit floating point values including:
a first plurality of bits stored in the memory, the first plurality of bits to form a mantissa of the respective floating point value;
a second plurality of bits stored in the memory, the second plurality of bits to form an exponent portion of the respective floating point value; and
a one-bit switch stored in the memory, the one-bit switch to selectively combine the exponent portion of the respective 16-bit floating point value with the five-bit shared exponent; and
adjust, in response to prediction of an overflow condition or an underflow condition during training of the neural network, one or more of the 16-bit floating point values in the memory to avoid the predicted overflow or underflow.
2. The system of claim 1 , wherein each of the plurality of 16-bit floating point values included in the neural network training tensor includes:
a six bit mantissa provided by the first plurality of bits;
an eight bit exponent provided by the second plurality of bits; and
a one-bit sign.
3. The system of claim 1 , wherein the mantissa is a variable bit-length mantissa, the exponent portion is a variable bit-length exponent, and wherein the instructions further cause the processor circuitry to select, based on one or more neural network parameters:
a first number of bits to represent the variable bit-length mantissa; and
a second number of bits to represent the variable bit-length exponent.
4. The system of claim 3 , wherein the one or more neural network parameters include a trend indicative of at least one of: an underflow condition or an overflow condition in one or more of the 16-bit floating point values included in the neural network training tensor.
5. The system of claim 1 , wherein each of the plurality of 16-bit floating point values included in the neural network training tensor includes:
the mantissa provided by the first plurality of bits;
the exponent portion provided by the second plurality of bits; and
a one-bit sign.
6. A method of training a neural network, comprising:
generating, by processor circuitry, a neural network training tensor that includes:
a five-bit shared exponent stored in a memory, the five-bit shared exponent common to each of a plurality of 16-bit floating point values included in the neural network training tensor, each of the 16-bit floating point values including:
a first plurality of bits stored in the memory, the first plurality of bits to form a mantissa of the respective floating point value; and
a second plurality of bits stored in the memory, the second plurality of bits form an exponent portion of the respective floating point value; and
a one-bit switch stored in the memory, the one-bit switch to selectively combine the exponent portion of the respective 16-bit floating point value with the five-bit shared exponent;
adjusting, by the processor circuitry in response to prediction of an overflow condition or an underflow condition during training of the neural network, one or more of the 16-bit floating point values in the memory to avoid the predicted overflow or underflow; and
providing, by the processor circuitry, at least one of the plurality of 16-bit floating point values included in the neural network training tensor as an input to the neural network.
7. The method of claim 6 , wherein the generating of the neural network training tensor includes:
generating the neural network training tensor that includes the plurality of 16-bit floating point values and the five-bit shared exponent, wherein each of the plurality of 16-bit floating point values includes:
a six bit mantissa provided by the first plurality of bits;
an eight bit exponent provided by the second plurality of bits; and
a one-bit sign.
8. The method of claim 6 , wherein the generating of the neural network training tensor includes:
generating the neural network training tensor that includes the plurality of 16-bit floating point values and the five-bit shared exponent, wherein each of the plurality of 16-bit floating point values includes:
a variable bit-length mantissa provided by the first plurality of bits;
a variable bit-length exponent provided by the second plurality of bits; and
a one-bit sign.
9. The method of claim 8 , further including selecting, by the processor circuitry based on one or more neural network parameters:
a first number of bits to represent the variable bit-length mantissa; and
a second number of bits to represent the variable bit-length exponent.
10. The method of claim 9 , further including:
detecting, by the processor circuitry, a trend in one or more of the plurality of 16-bit floating point values included in the training tensor, the detected trend indicative of at least one of: an underflow condition or an overflow condition in one or more of the 16-bit floating point values included in the neural network training tensor.
11. The method of claim 9 , wherein the selecting of the first number of bits to represent the variable bit-length mantissa and the second number of bits to represent the variable bit-length exponent includes:
selecting, by the processor circuitry, a first number of bits included in the first plurality of bits and a second number of bits included in the second plurality of bits responsive to detecting a trend indicative of at least one of: an underflow condition or an overflow condition in one or more of the 16-bit floating point values included in the neural network training tensor.
12. The method of claim 6 , wherein the generating of the neural network training tensor includes:
generating the neural network training tensor that includes the plurality of 16-bit floating point values and the five-bit shared exponent, wherein each of the plurality of 16-bit floating point values includes:
the mantissa provided by the first plurality of bits;
the exponent portion provided by the second plurality of bits; and
a one-bit sign.
13. A neural network training system, comprising:
means for generating a neural network training tensor that includes:
a five-bit shared exponent stored in a memory, the five-bit shared exponent common to each of a plurality of 16-bit floating point values included in the neural network training tensor, each of the 16-bit floating point values including:
a first plurality of bits stored in the memory, the first plurality of bits to form a mantissa of the respective floating point value;
a second plurality of bits stored in the memory, the second plurality of bits to form an exponent portion of the respective floating point value; and
a one-bit switch stored in the memory, the one-bit switch to selectively combine the exponent portion of the respective 16-bit floating point value with the five-bit shared exponent;
means for adjusting, in response to prediction of an overflow condition or an underflow condition during training of a neural network, one or more of the 16-bit floating point values in the memory to avoid the predicted overflow or underflow; and
means for providing as an input to the neural network at least one of the plurality of 16-bit floating point values included in the neural network training tensor.
14. The system of claim 13 , wherein the means for generating the neural network training tensor includes:
means for generating the neural network training tensor that includes the plurality of 16-bit floating point values and the five-bit shared exponent, wherein each of the plurality of 16-bit floating point values includes:
a six bit mantissa provided by the first plurality of bits;
an eight bit exponent provided by the second plurality of bits; and
a one-bit sign.
15. The system of claim 13 , wherein the means for generating the neural network training tensor further includes:
means for generating the neural network training tensor that includes the plurality of 16-bit floating point values and the five-bit shared exponent, wherein each of the plurality of 16-bit floating point values includes:
a variable bit-length mantissa provided by the first plurality of bits;
a variable bit-length exponent provided by the second plurality of bits; and
a one-bit sign.
16. The system of claim 15 , further including means for selecting:
a first number of bits to represent the variable bit-length mantissa; and
a second number of bits to represent the variable bit-length exponent.
17. The system of claim 16 , further including means for detecting a trend in one or more of the plurality of 16-bit floating point values included in the neural network training tensor, the detected trend indicative of at least one of: a potential underflow condition or a potential overflow condition in one or more of the 16-bit floating point values included in the neural network training tensor.
18. The system of claim 16 , wherein the means for selecting a first number of bits to represent the variable bit-length mantissa and a second number of bits to represent the variable bit-length exponent further includes means for selecting a first number of bits included in the first plurality of bits and a second number of bits included in the second plurality of bits responsive to detecting a trend indicative of at least one of: an underflow condition or an overflow condition in one or more of the 16-bit floating point values included in the neural network training tensor.
19. The system of claim 13 , wherein the means for generating the neural network training tensor further includes means for generating the neural network training tensor that includes the plurality of 16-bit floating point values and the five-bit shared exponent, wherein each of the plurality of 16-bit floating point values includes:
the mantissa provided by the first plurality of bits;
the exponent portion provided by the second plurality of bits; and
a one-bit sign.
20. A non-transitory, computer-readable storage medium that includes machine readable instructions that, when executed by processor circuitry, cause the processor circuitry to:
generate a neural network training tensor that includes:
a five-bit shared exponent stored in a memory, the five-bit shared exponent common to each of a plurality of 16-bit floating point values included in the neural network training tensor, each of the plurality of 16-bit floating point values including:
a first plurality of bits stored in the memory, the first plurality of bits to form a mantissa of the respective floating point value;
a second plurality of bits stored in the memory, the second plurality of bits to form an exponent portion of the respective floating point value; and
a one-bit switch stored in the memory, the one-bit switch to selectively combine the exponent portion of the respective 16-bit floating point value with the five-bit shared exponent; and
adjust, in response to a prediction of an overflow or underflow condition during training of a neural network, one or more of the 16-bit floating point values in the memory to avoid the predicted overflow or underflow.
21. The non-transitory, computer-readable storage medium of claim 20 , wherein each of the plurality of 16-bit floating point values includes:
a six bit mantissa provided by the first plurality of bits;
an eight bit exponent provided by the second plurality of bits; and
a one-bit sign.
22. The non-transitory, computer-readable storage medium of claim 20 , wherein the mantissa is a variable bit-length mantissa, the exponent portion is a variable bit-length exponent, and wherein the instructions, when executed, cause the processor circuitry to select, based on one or more neural network parameters:
a first number of bits to represent the variable bit-length mantissa; and
a second number of bits to represent the variable bit-length exponent.
23. The non-transitory, computer-readable storage medium of claim 22 , wherein the one or more neural network parameters indicate an underflow condition or an overflow condition in one or more of the plurality of 16-bit floating point values included in the neural network training tensor.
24. The non-transitory, computer-readable storage medium of claim 20 , wherein the machine readable instructions that cause the processor circuitry to generate the neural network training tensor that includes a plurality of 16-bit floating point values further cause the processor circuitry to:
generate a neural network training tensor that includes a plurality of 16-bit floating point values, each of the plurality of 16-bit floating point values including:
the mantissa provided by the first plurality of bits;
the exponent portion provided by the second plurality of bits; and
a one-bit sign.
25. Processor circuitry programmed to implement a tensor data storage structure in a plurality of hardware registers, comprising:
a five-bit shared hardware register to store a first exponent value, the five-bit shared hardware register associated with each respective one of a plurality of 16-bit floating point hardware registers, each of the plurality of 16-bit floating point hardware registers including:
a first hardware register to store a mantissa value;
a second hardware register to store a second exponent value;
a third hardware register to store a sign bit; and
a one-bit switch hardware register to selectively combine each respective 16-bit floating point hardware register with the five-bit shared exponent hardware register; and
predictive circuitry to adjust one or more of the 16-bit floating point hardware registers in response to a prediction of an overflow or underflow condition in one or more of the 16-bit floating point hardware registers to avoid the overflow or underflow.