Incremental precision networks using residual inference and fine-grain quantization
One embodiment provides for a computer-readable medium storing instructions that cause one or more processors to perform operations comprising determining a per-layer scale factor to apply to tensor data associated with layers of a neural network model and converting the tensor data to converted tensor data. The tensor data may be converted from a floating point datatype to a second datatype that is an 8-bit datatype. The instructions further cause the one or more processors to generate an output tensor based on the converted tensor data and the per-layer scale factor.
1. A graphics processor comprising:
a memory interface; and
a processing cluster coupled with the memory interface, the processing cluster including a plurality of processing resources coupled via a data interconnect, a processing resource of the plurality of processing resources including:
first circuitry including a weight ternarization circuit and an activation quantization circuit, the weight ternarization circuit to convert a weight tensor from a floating-point representation to a ternary representation including a two-bit ternary weight and an eight-bit integer scale factor, and the two-bit ternary weight to represent a weight value of one of negative one, zero, or positive one and wherein the activation quantization circuit is to convert an activation tensor from a floating-point representation to an integer representation; and
second circuitry configured to perform a set of parallel integer compute operations on the ternary representation of the weight tensor and the integer representation of the activation tensor.
2. The graphics processor as in claim 1 , the second circuitry including one or more ternary logic units to perform one or more operations in the set of parallel integer compute operations.
3. The graphics processor as in claim 2 , wherein the second circuitry is configured to perform the set of parallel integer compute operations in response to an instruction provided by a machine learning inferencing framework.
4. The graphics processor as in claim 1 , the activation quantization circuit to convert the activation tensor from a single-precision floating-point representation to an eight-bit integer representation.
5. The graphics processor as in claim 1 , the weight ternarization circuit to convert the weight tensor from a single-precision floating-point representation to the ternary representation.
6. The graphics processor as in claim 5 , wherein the eight-bit integer scale factor is determined to minimize L2 loss between values of a group of pre-trained weights and a group of ternary weights.
7. The graphics processor as in claim 5 , wherein to convert the weight tensor from the single-precision floating-point representation, the weight ternarization circuit is to decompose the weight tensor into a set of orthogonal vectors and ternarize components of the set of orthogonal vectors.
8. The graphics processor as in claim 1 , the weight ternarization circuit to ternarize first weights having a first data distribution into first ternarized weights having a first scale factor and ternarize second weights having a second data distribution into second ternarized weights having a second scale factor different from the first scale factor.
9. A method comprising:
on a graphics processor comprising a processing cluster including a plurality of processing resources coupled via a data interconnect:
converting a weight tensor from a floating-point representation to a ternary representation including a two-bit ternary weight and an eight-bit integer scale factor using a weight ternarization circuit, the two-bit ternary weight to represent a weight value of one of negative one, zero, or positive one;
converting an activation tensor from a floating-point representation to an integer representation using an activation quantization circuit; and
performing a set of parallel integer compute operations on the ternary representation of the weight tensor and the integer representation of the activation tensor using second circuitry.
10. The method as in claim 9 , further comprising performing one or more operations in the set of parallel integer compute operations via one or more ternary logic units of the second circuitry.
11. The method as in claim 10 , further comprising performing the set of parallel integer compute operations in response to an instruction provided by a machine learning inferencing framework.
12. The method as in claim 9 , further comprising converting the activation tensor from a single-precision floating-point representation to an eight-bit integer representation.
13. The method as in claim 9 , further comprising converting the weight tensor from a single-precision floating-point representation to the ternary representation.
14. The method as in claim 13 , further comprising determining the eight-bit integer scale factor to minimize L2 loss between values of a group of pre-trained weights and a group of ternary weights.
15. The method as in claim 14 , wherein converting the weight tensor from the single-precision floating-point representation includes decomposing the weight tensor into a set of orthogonal vectors and ternarizing components of the set of orthogonal vectors.
16. The method as in claim 9 , further comprising, via the weight ternarization circuit:
ternarizing first weights having a first data distribution into first ternarized weights having a first scale factor; and
ternarizing second weights having a second data distribution into second ternarized weights having a second scale factor different from the first scale factor.
17. A data processing system comprising:
a memory device; and
graphics processor coupled with the memory device via a memory interface, the graphics processor comprising a processing cluster including a plurality of processing resources coupled via a data interconnect, wherein a processing resource of the plurality of processing resources includes:
first circuitry including a weight ternarization circuit and an activation quantization circuit, the weight ternarization circuit to convert a weight tensor from a floating-point representation to a ternary representation including a two-bit ternary weight and an eight-bit integer scale factor, and the two-bit ternary weight to represent a weight value of one of negative one, zero, or positive one and wherein the activation quantization circuit is to convert an activation tensor from a floating-point representation to an integer representation; and
second circuitry configured to perform a set of parallel integer compute operations on the ternary representation of the weight tensor and the integer representation of the activation tensor.
18. The data processing system as in claim 17 , the second circuitry including one or more ternary logic units to perform one or more operations in the set of parallel integer compute operations.
19. The data processing system as in claim 18 , wherein the second circuitry is configured to perform the set of parallel integer compute operations in response to an instruction provided by a machine learning inferencing framework.
20. The data processing system as in claim 17 , the activation quantization circuit to convert the activation tensor from a single-precision floating-point representation to an eight-bit integer representation.