Neural network processing using mixed-precision data representation
Techniques for neural network processing using mixed-precision data representation are disclosed. Access to a processor that supports single-precision operations is obtained, where the processor is used for neural network calculations. A first input data element and a second input data element are presented for manipulation on the processor, where the manipulation supports the neural network calculations. The first input data element is manipulated with the second input data element using the processor, where the first input data element comprises a 16-bit reduced floating point representation. A result of the manipulation is output, where the result comprises a single-precision data representation element. The result is forwarded to a next layer of the neural network, based on the outputting.
1 . A processor-implemented method for data manipulation comprising:
accessing a first input data element and a second input data element for manipulation on a processor that supports single precision operations, wherein:
the first input data element comprises one or more parameters of a neural network trained on training data represented by a single-precision data representation, wherein each parameter of the neural network comprises a 16-bit reduced floating point representation that does not vary from parameter to parameter of the neural network, and wherein training the neural network using the single-precision training data produces the parameters of the neural network as represented by a 16-bit reduced floating point representation; and
the second input data element comprises input data, that does not define any parameter of the neural network, for processing by the neural network;
converting the second input data element from a first data representation to a 16-bit reduced floating point representation;
processing, by at least a first layer of the neural network, the first and second input data elements, comprising:
manipulating, by the processor, the first input data element with the second input data element, wherein the first input data element comprises a 16-bit reduced floating point representation, to produce a single-precision data output as a result of the manipulation of the 16-bit reduced floating point first input data element and the 16-bit reduced floating point second input data element;
outputting the result of the manipulation, wherein the result comprises a single-precision data representation element; and
forwarding the result to a next layer of the neural network, based on the outputting;
determining whether the next layer of the neural network is an output layer of the neural network;
in response to a determination that the next layer is not the output layer of the neural network, then defining the second input data element as the result of the manipulation and repeating the accessing, converting, and processing steps with the next layer of the neural network; and
in response to a determination that the next layer is the output layer, then outputting the result of the manipulation.
2 . The method of claim 1 wherein the second input data element comprises a 16-bit reduced floating point representation.
3 . The method of claim 1 wherein the second input data element comprises an 8-bit integer representation element.
4 . The method of claim 3 further comprising factoring the 8-bit integer representation element into a 16-bit reduced floating point representation element to facilitate the manipulating.
5 . The method of claim 3 wherein the 8-bit integer representation comprises image data.
6 . The method of claim 5 wherein the image data uses an 8-bit unsigned integer RGB representation.
7 . The method of claim 1 wherein the first input data element comprises one or more neural network weights.
8 . The method of claim 7 wherein the weights are trained using single-precision data representations.
9 . The method of claim 1 wherein the second input data element comprises a single-precision data representation element.
10 . The method of claim 1 wherein the neural network processing comprises image processing.
11 . The method of claim 1 wherein the next layer is a bottleneck layer of the neural network.
12 . The method of claim 1 wherein the manipulating comprises neural network addition or subtraction operations.
13 . The method of claim 12 wherein the addition or subtraction operations include a single-precision data representation operand and a 16-bit reduced floating point data representation operand as inputs resulting in a single-precision data representation output.
14 . The method of claim 12 wherein the addition or subtraction operations facilitate calculating a dot product.
15 . The method of claim 14 wherein the dot product includes an 8-bit integer data representation operand and a 16-bit reduced floating point data representation operand as inputs resulting in a single-precision data representation output.
16 . The method of claim 1 further comprising including commands, for the processor, that support single-precision operands and 16-bit reduced floating point representation operands in the same operation.
17 . The method of claim 1 further comprising adding commands for the processor that support 8-bit integer representation operands and 16-bit reduced floating point representation operands in the same operation.
18 . The method of claim 1 wherein the 16-bit reduced floating point representation comprises a bfloat16 data representation.
19 . The method of claim 1 wherein the 16-bit reduced floating point representation comprises a reduced mantissa floating point representation.
20 . A computer program product embodied in a non-transitory computer readable medium for data manipulation, the computer program product comprising code which causes one or more processors to perform operations of:
accessing a first input data element and a second input data element for manipulation on a processor that supports single precision operations, wherein:
the first input data element comprises one or more parameters of a neural network trained on training data represented by a single-precision data representation, wherein each parameter of the neural network comprises a 16-bit reduced floating point representation that does not vary from parameter to parameter of the neural network, and wherein training the neural network using the single-precision training data produces the parameters of the neural network as represented by a 16-bit reduced floating point representation; and
the second input data element comprises input data, that does not define any parameter of the neural network, for processing by the neural network;
converting the second input data element from a first data representation to a 16-bit reduced floating point representation;
processing, by at least a first layer of the neural network, the first and second input data elements, comprising:
manipulating, by the processor, the first input data element with the second input data element, wherein the first input data element comprises a 16-bit reduced floating point representation, to produce a single-precision data output as a result of the manipulation of the 16-bit reduced floating point first input data element and the 16-bit reduced floating point second input data element;
outputting the result of the manipulation, wherein the result comprises a single-precision data representation element; and
forwarding the result to a next layer of the neural network, based on the outputting;
determining whether the next layer of the neural network is an output layer of the neural network;
in response to a determination that the next layer is not the output layer of the neural network, then defining the second input data element as the result of the manipulation and repeating the accessing, converting, and processing steps with the next layer of the neural network; and
in response to a determination that the next layer is the output layer, then outputting the result of the manipulation.
21 . A computer system for data manipulation comprising: a memory which stores instructions; and one or more processors attached to the memory wherein the one or more processors, when executing the instructions which are stored, are configured to:
access a first input data element and a second input data element for manipulation on a processor that supports single precision operations, wherein:
the first input data element comprises one or more parameters of a neural network trained on training data represented by a single-precision data representation, wherein each parameter of the neural network comprises a 16-bit reduced floating point representation that does not vary from parameter to parameter of the neural network, and wherein training the neural network using the single-precision training data produces the parameters of the neural network as represented by a 16-bit reduced floating point representation; and
the second input data element comprises input data that does not define any parameter of the neural network, for processing by the neural network;
convert the second input data element from a first data representation to a 16-bit reduced floating point representation;
process, by at least a first layer of the neural network, the first and second input data elements, comprising:
manipulating, by the processor, the first input data element with the second input data element, wherein the first input data element comprises a 16-bit reduced floating point representation, to produce a single-precision data output as a result of the manipulation of the 16-bit reduced floating point first input data element and the 16-bit reduced floating point second input data element;
outputting the result of the manipulation, wherein the result comprises a single-precision data representation element; and
forwarding the result to a next layer of the neural network, based on the outputting;
determine whether the next layer of the neural network is an output layer of the neural network;
in response to a determination that the next layer is not the output layer of the neural network, then define the second input data element as the result of the manipulation and repeat the accessing, converting, and processing steps with the next layer of the neural network; and
in response to a determination that the next layer is the output layer, then output the result of the manipulation.