Incremental precision networks using residual inference and fine-grain quantization

1. A non-transitory computer-readable medium storing instructions, execution of which by one or more processors including a general-purpose graphics processor, causes the one or more processors to perform operations comprising:

determining a first scale factor to apply to first tensor data associated with a first layer of a neural network model;

converting the first tensor data associated with the first layer of the neural network model to generate first converted tensor data, the first tensor data converted from a first datatype to a second datatype, wherein the second datatype is an 8-bit datatype;

generating a first output tensor that is associated with the first layer of the neural network model, the first output tensor generated via the general-purpose graphics processor based on the first converted tensor data and the first scale factor;

determining a second scale factor to apply to second tensor data associated with a second layer of the neural network model;

converting the second tensor data associated with the second layer of the neural network model to generate second converted tensor data, the second tensor data converted from the first datatype to the second datatype; and

generating a second output tensor that is associated with the second layer of the neural network model, the second output tensor generated via the general-purpose graphics processor based on the second converted tensor data and the second scale factor.

2. The non-transitory computer-readable medium of claim 1 , wherein the first datatype is a floating-point datatype.

3. The non-transitory computer-readable medium of claim 2 , wherein the first datatype is a 32-bit floating-point datatype.

4. The non-transitory computer-readable medium of claim 2 , wherein the first scale factor is determined to reduce a difference between values of the first output tensor when generated via the first tensor data relative to the values of the first output tensor when generated via the first converted tensor data.

5. The non-transitory computer-readable medium of claim 2 , wherein the second scale factor is determined to reduce a difference between values of the second output tensor when generated via the second tensor data relative to the values of the second output tensor when generated via the second converted tensor data.

6. The non-transitory computer-readable medium of claim 1 , wherein the first tensor data includes weight values associated with the first layer of the neural network model.

7. The non-transitory computer-readable medium of claim 6 , wherein the second tensor data includes weight values associated with the second layer of the neural network model.

8. The non-transitory computer-readable medium of claim 7 , the operations further comprising:

converting a first input activation tensor associated with the first layer of the neural network model to generate first converted input activation tensor; and

generating the first output tensor based in part on the first converted input activation tensor.

9. The non-transitory computer-readable medium of claim 8 , the operations further comprising generating the second output tensor based at least in part on the first output tensor and the second converted tensor data.

10. The non-transitory computer-readable medium of claim 9 , the operations further comprising:

generating the first output tensor in the first datatype; and

converting the first output tensor from the first datatype to the second datatype before generating the second output tensor.

11. An apparatus comprising:

a memory device; and

one or more processors coupled with the memory device, the one or more processors including a general-purpose graphics processor, the one or more processors comprising circuitry configured to:

determine a first scale factor to apply to first tensor data associated with a first layer of a neural network model;

convert the first tensor data associated with the first layer of the neural network model to generate first converted tensor data, the first tensor data converted from a first datatype to a second datatype, wherein the second datatype is an 8-bit datatype;

generate a first output tensor that is associated with the first layer of the neural network model, the first output tensor generated via the general-purpose graphics processor based on the first converted tensor data and the first scale factor;

determine a second scale factor to apply to second tensor data associated with a second layer of the neural network model;

convert the second tensor data associated with the second layer of the neural network model to generate second converted tensor data, the second tensor data converted from the first datatype to the second datatype; and

generate a second output tensor that is associated with the second layer of the neural network model, the second output tensor generated via the general-purpose graphics processor based on the second converted tensor data and the second scale factor.

12. The apparatus of claim 11 , wherein the first datatype is a floating-point datatype.

13. The apparatus of claim 12 , wherein the first datatype is a 32-bit floating-point datatype.

14. The apparatus of claim 12 , wherein the first scale factor is determined to reduce a difference between values of the first output tensor when generated via the first tensor data relative to the values of the first output tensor when generated via the first converted tensor data.

15. The apparatus of claim 12 , wherein the second scale factor is determined to reduce a difference between values of the second output tensor when generated via the second tensor data relative to the values of the second output tensor when generated via the second converted tensor data.

16. The apparatus of claim 11 , wherein the first tensor data includes weight values associated with the first layer of the neural network model.

17. The apparatus of claim 16 , wherein the second tensor data includes weight values associated with the second layer of the neural network model.

18. The apparatus of claim 17 , the circuitry configured to:

convert a first input activation tensor associated with the first layer of the neural network model to generate first converted input activation tensor; and

generate the first output tensor based in part on the first converted input activation tensor.

19. The apparatus of claim 18 , the circuitry configured to generate the second output tensor based at least in part on the first output tensor and the second converted tensor data.

20. The apparatus of claim 19 , the circuitry configured to:

generate the first output tensor in the first datatype; and

convert the first output tensor from the first datatype to the second datatype before generating the second output tensor.

21. A general-purpose graphics processor comprising:

a system interface; and

a machine learning accelerator circuitry coupled with the system interface, the machine learning accelerator circuitry configured to:

convert first tensor data associated with a first layer of a neural network model to generate first converted tensor data, the first tensor data converted from a first datatype to a second datatype, wherein the second datatype is an 8-bit datatype;

generate a first output tensor that is associated with the first layer of the neural network model, the first output tensor generated via the general-purpose graphics processor based on the first converted tensor data and a first scale factor, the first scale factor determined for the first tensor data;

convert second tensor data associated with a second layer of the neural network model to generate second converted tensor data, the second tensor data converted from the first datatype to the second datatype; and

generate a second output tensor that is associated with the second layer of the neural network model, the second output tensor generated via the general-purpose graphics processor based on the second converted tensor data and a second scale factor, the second scale factor determined for the second tensor data.

22. The general-purpose graphics processor of claim 21 , wherein the first datatype is a floating-point datatype.

23. The general-purpose graphics processor of claim 22 , wherein the first datatype is a 32-bit floating-point datatype.

24. The general-purpose graphics processor of claim 22 , wherein the first scale factor is determined to reduce a difference between values of the first output tensor when generated via the first tensor data relative to the values of the first output tensor when generated via the first converted tensor data.

25. The general-purpose graphics processor of claim 22 , wherein the second scale factor is determined to reduce a difference between values of the second output tensor when generated via the second tensor data relative to the values of the second output tensor when generated via the second converted tensor data.