Scaling half-precision floating point tensors for training deep neural networks
A graphics processor is described that includes a single instruction, multiple thread (SIMT) architecture including hardware multithreading. The multiprocessor can execute parallel threads of instructions associated with a command stream, where the multiprocessor includes a set of functional units to execute at least one of the parallel threads of the instructions. The set of functional units can include a mixed precision tensor processor to perform tensor computations to generate loss data. The loss data is stored as a floating-point data type and scaled by a scaling factor to enable a data distribution of a gradient tensor generated based on the loss data to be represented by a 16-bit floating point data type.
1. A graphics processor comprising:
a memory controller;
a level-two (L2) cache memory coupled with the memory controller; and
a multiprocessor coupled with the memory controller, the multiprocessor having a single instruction, multiple thread (SIMT) architecture including hardware multithreading, the multiprocessor to execute parallel threads of instructions associated with a command stream, the multiprocessor including a set of functional units to execute at least one of the parallel threads of the instructions, wherein the set of functional units include a mixed precision tensor processor to perform tensor computations to generate loss data, wherein the loss data is stored as a floating-point data type and scaled by a scaling factor to enable a data distribution of a gradient tensor generated based on the loss data to be represented by a 16-bit floating point data type.
2. The graphics processor as in claim 1 , wherein the mixed precision tensor processor is to perform tensor computations associated with a layer of a neural network to generate loss data.
3. The graphics processor as in claim 2 , wherein the mixed precision tensor processor is additionally to perform a backward propagation operation associated with the layer of the neural network.
4. The graphics processor as in claim 3 , wherein the multiprocessor is to dynamically adjust the scaling factor based on values within the gradient tensor.
5. The graphics processor as in claim 1 , wherein to scale the loss data includes to increase a minimum value of the gradient tensor to above the minimum value representable by the 16-bit floating point data type.
6. The graphics processor as in claim 5 , wherein to scale the loss data includes to select a scaling factor such that the maximum value of the gradient tensor is below the maximum value representable by the 16-bit floating point data type.
7. The graphics processor as in claim 1 , additionally comprising a 3D memory stack coupled to the memory controller.
8. The graphics processor as in claim 7 , wherein the 3D memory stack includes high-bandwidth memory.
9. The graphics processor as in claim 1 , additionally comprising a graphics memory coupled with the memory controller of the graphics processor, wherein the graphics memory includes a graphics double data rate (GDDR) memory including GDDR6 memory.
10. The graphics processor as in claim 1 , wherein the 16-bit floating point data type is a non-IEEE floating point data type.
11. A non-transitory machine-readable medium to store instructions to cause one or more processors to perform operations comprising:
executing, on a graphics processor, parallel threads of instructions associated with a command stream via a multiprocessor coupled with a memory controller, the multiprocessor having a single instruction, multiple thread (SIMT) architecture including hardware multithreading, wherein the multiprocessor includes a set of functional units to execute at least one of the parallel threads of the instructions, the set of functional units including a mixed precision tensor processor;
performing, via the mixed precision tensor processor, tensor computations associated with a layer of a neural network to generate loss data, the loss data stored in a floating-point format;
scaling the loss data generated via the tensor computations by a scaling factor to enable a data distribution of a gradient tensor generated based on the loss data to be represented by a scaled gradient tensor, the scaled gradient tensor stored as a 16-bit floating point data type; and
performing a backward propagation operation associated with the layer of the neural network to generate the scaled gradient tensor.
12. The non-transitory machine-readable medium as in claim 11 , the operations additionally comprising dynamically adjusting the scaling factor based on output of the tensor computations associated with the layer of a neural network.
13. The non-transitory machine-readable medium as in claim 12 , the operations additionally comprising dynamically adjusting the scaling factor based on values within the gradient tensor.
14. The non-transitory machine-readable medium as in claim 13 , the operations additionally comprising selecting a scaling factor to increase a minimum value of the gradient tensor to above the minimum value representable by the 16-bit floating point data type.
15. The non-transitory machine-readable medium as in claim 14 , the operations additionally comprising selecting a scaling factor to scale the maximum value of the gradient tensor to below the maximum value representable by the 16-bit floating point data type.
16. The non-transitory machine-readable medium as in claim 11 , wherein the 16-bit floating point data type is a non-IEEE floating point data type.
17. A graphics processing system comprising:
a graphics processor including a memory controller, a level-two (L2) cache memory coupled with the memory controller, and a multiprocessor coupled with the memory controller, the multiprocessor having a single instruction, multiple thread (SIMT) architecture including hardware multithreading, the multiprocessor to execute parallel threads of instructions associated with a command stream, the multiprocessor including a set of functional units to execute at least one of the parallel threads of the instructions, wherein the set of functional units include a mixed precision tensor processor and are configured to scale loss data generated via tensor computations performed on by the mixed precision tensor processor on a layer of a neural network, the loss data is stored as a floating-point data type, and the loss data is scaled by a scaling factor to enable a data distribution of a gradient tensor generated based on the loss data to be represented by a 16-bit floating point data type; and
a graphics memory coupled with the memory controller of the graphics processor.
18. The graphics processing system as in claim 17 , additionally comprising a host interconnect bus coupled to the graphics processor and the graphics memory.
19. The graphics processing system as in claim 17 , wherein the graphics memory includes a graphics double data rate (GDDR) memory including GDDR6 memory.
20. The graphics processing system as in claim 17 , wherein the mixed precision tensor processor is to perform tensor computations associated with a layer of a neural network to generate loss data.
21. The graphics processing system as in claim 20 , wherein the mixed precision tensor processor is additionally to perform a backward propagation operation associated with the layer of the neural network.
22. The graphics processing system as in claim 21 , wherein the multiprocessor is to dynamically adjust the scaling factor based on values within the gradient tensor.
23. The graphics processing system as in claim 22 , wherein to scale the loss data includes to select a scaling factor such that the maximum value of the gradient tensor is below the maximum value representable by the 16-bit floating point data type and increase a minimum value of the gradient tensor to above the minimum value representable by the 16-bit floating point data type.
24. The graphics processing system as in claim 17 , wherein the 16-bit floating point data type is a non-IEEE floating point data type.