Scaling half-precision floating point tensors for training deep neural networks

1. A machine-learning accelerator comprising:

a multiprocessor to execute parallel threads of an instruction stream, the multiprocessor including a compute unit, the compute unit including a set of functional units, each functional unit to execute at least one of the parallel threads of the instruction stream; and

wherein the compute unit includes compute logic configured to execute a single instruction to scale an input tensor associated with a layer of a neural network according to a scale factor, the input tensor stored in a floating-point data type, the compute logic to scale the input tensor to enable a data distribution of data of the input tensor to be represented by a 16-bit floating point data type.

2. The machine-learning accelerator as in claim 1 , the compute logic to compute an exponent bias based on an absolute maximum value of the input tensor and a dynamic range of the 16-bit floating point data type.

3. The machine-learning accelerator as in claim 2 , the compute logic to convert weights and activations of the layer of the neural network to scaled 16-bit floating-point tensors.

4. The machine-learning accelerator as in claim 3 , the compute logic to perform a compute operation on the scaled 16-bit floating-point tensors, the compute operation including one a multiply, add, or a fused multiply-add.

5. The machine-learning accelerator as in claim 4 , the compute logic to generate a set of 32-bit intermediate results in response to the compute operation on the scaled 16-bit floating-point tensors.

6. The machine-learning accelerator as in claim 5 , the compute logic to re-scale the set of 32-bit intermediate results using the exponent bias and down-convert the set of 32-bit intermediate results into scaled 16-bit floating point values.

7. The machine-learning accelerator as in claim 1 , the compute logic to perform a compute operation on scaled 16-bit floating-point tensors to generate intermediate values, determine if the intermediate values are close to saturation, and re-scale the intermediate values when the intermediate values are close to saturation.

8. The machine-learning accelerator as in claim 7 , the compute logic to update an exponent bias value after the re-scale of the intermediate values.

9. A method implemented via a machine-learning accelerator, the method comprising:

executing parallel threads of a parallel instruction stream, the parallel instruction stream including at least one single instruction to perform a scaled tensor compute operation;

in response to the single instruction, scaling data of an input tensor associated with a layer of a neural network according to a scale factor, executing one or more compute operations on scaled data of the input tensor, and re-scaling computed data of the compute operations to generate re-scaled computed data;

down-converting the re-scaled computed data; and

storing down-converted re-scaled computed data to a 16-bit floating-point data type.

10. The method as in claim 9 , additionally comprising computing an exponent bias based on an absolute maximum value of the input tensor and a dynamic range of the 16-bit floating-point data type and scaling the data of the input tensor using the exponent bias.

11. The method as in claim 10 , additionally comprising converting weights and activations of the layer of the neural network to scaled 16-bit floating-point tensors.

12. The method as in claim 11 , wherein executing the one or more compute operations on the scaled data of the input tensor includes executing a multiply, add, or a fused multiply-add operation.

13. The method as in claim 12 , wherein computed data of the compute operations include a set of 32-bit intermediate results.

14. The method as in claim 13 , additionally comprising re-scaling the set of 32-bit intermediate results using the exponent bias and down-converting the set of 32-bit intermediate results into scaled 16-bit floating-point values.

15. The method as in claim 9 , additionally comprising:

performing a compute operation on scaled 16-bit floating-point tensors to generate intermediate values;

determining if the intermediate values are close to saturation;

re-scaling the intermediate values in response to determining that the intermediate values are close to saturation; and

updating an exponent bias value after re-scaling the intermediate values.

16. A data processing system comprising:

a memory device; and

a general-purpose graphics processing unit including an instruction decoder to decode a single instruction including multiple operands into a single decoded instruction, at least one of the multiple operands associated with input tensor data of a layer of a neural network, and a compute unit including compute logic configured to execute the single instruction to scale data of the input tensor to enable a data distribution of data of the input tensor to be represented by a 16-bit floating point data type.

17. The data processing system as in claim 16 , the compute logic to:

compute an exponent bias based on an absolute maximum value of the input tensor and a dynamic range of the 16-bit floating point data type;

convert weights and activations of the layer of the neural network to scaled 16-bit floating-point tensors;

perform a compute operation on the scaled 16-bit floating-point tensors;

generate a set of 32-bit intermediate results in response to the compute operation

on the scaled 16-bit floating-point tensors;

re-scale the set of 32-bit intermediate results using the exponent bias; and

down-convert the set of 32-bit intermediate results into scaled 16-bit floating point values.

18. The data processing system as in claim 17 , the compute operation including one a multiply, add, or a fused multiply-add.

19. The data processing system as in claim 16 , the compute logic to perform a compute operation on scaled 16-bit floating-point tensors to generate intermediate values, determine if the intermediate values are close to saturation, and re-scale the intermediate values when the intermediate values are close to saturation.

20. The data processing system as in claim 19 , the compute logic to update an exponent bias value after the re-scale of the intermediate values.