Techniques for conformance testing computational operations
Examples described herein generally relate to performing conformance testing of a computational operation. A reference result including one or more reference intermediate products and a reference accumulator output at a first level of precision can be generated for the computational operation and based on one or more inputs. A hardware result can similarly be created using hardware at a second level of precision. The reference result can be compared to the hardware result to determine a variance value. A conformance result can be output based on whether the variance value is within a threshold range.
1. A computer-implemented method for performing conformance testing of a machine learning (ML) computational operation configured to be implemented in a graphics processing unit (GPU) wherein the ML computational operation is performed by an algorithm specific to a hardware device, comprising:
generating, for the ML computational operation and based on one or more inputs, a reference result including one or more reference intermediate products and a reference accumulator output at a first level of precision, wherein the one or more reference intermediate products of the ML computational operation are within one or more ranges including a range where all values within the range are within a same single floating point exponent;
generating, for the ML computational operation and based on specifying, to the GPU, the one or more inputs and a second level of precision, a result for the hardware device including one or more hardware intermediate products and a hardware accumulator output using the GPU at the second level of precision; and
outputting a conformance result based on whether a variance between the reference result and the result is within a threshold range.
2. The computer-implemented method of claim 1 , wherein determining the one or more inputs comprises determining the one or more inputs as in the range of 1.0 and a square root of 2.0.
3. The computer-implemented method of claim 1 , wherein the one or more ranges include a first range with an exponent of zero, a second range with a minimum exponent, and a third range with a maximum exponent, and wherein generating the reference result, generating the result, and comparing the reference result to the generated result are performed for each of the one or more ranges.
4. The computer-implemented method of claim 1 , wherein the conformance result includes an indication of whether the generated result passes a conformance test.
5. The computer-implemented method of claim 1 , wherein generating the reference result, generating the result, and comparing the reference result to the generated result are performed for multiple configurations for the ML computational operation to output conformance results for each of the configurations.
6. The computer-implemented method of claim 5 , wherein the multiple configurations include:
a first configuration having a single precision to represent values of the one or more inputs, a single precision to represent values of the one or more hardware intermediate products, a single precision to represent values of an accumulator that generates the hardware accumulator output, and a single precision to represent values of the hardware accumulator output;
a second configuration having a half precision to represent values of the one or more inputs, a half precision to represent values of the one or more hardware intermediate products, a half precision to represent values of an accumulator that generates the hardware accumulator output, and a half precision to represent values of the hardware accumulator output;
a third configuration having a half precision to represent values of the one or more inputs, a half precision to represent values of the one or more hardware intermediate products, a full precision to represent values of an accumulator that generates the hardware accumulator output, and a half precision to represent values of the hardware accumulator output; and
a fourth configuration having a half precision to represent values of the one or more inputs, a full precision to represent values of the one or more hardware intermediate products, a full precision to represent values of an accumulator that generates the hardware accumulator output, and a half precision to represent values of the hardware accumulator output.
7. The computer-implemented method of claim 1 , wherein comparing the reference result to the generated result comprises comparing each of the one or more reference intermediate products to a corresponding one of the one or more hardware intermediate products and comparing the reference accumulator output to the hardware accumulator output to determine associated error values.
8. The computer-implemented method of claim 1 , wherein the ML computational operation includes a convolution operation.
9. A computing device for performing conformance testing of a machine learning (ML) computational operation configured to be implemented in a graphics processing unit (GPU) wherein the ML computational operation is performed by an algorithm specific to a hardware device, comprising:
a memory storing one or more parameters or instructions for developing an application; and
at least one processor coupled to the memory, wherein the at least one processor is configured to:
generate, for the ML computational operation and based on one or more inputs, a reference result including one or more reference intermediate products and a reference accumulator output at a first level of precision, wherein the one or more reference intermediate products of the ML computational operation are within one or more ranges including a range where all values within the range are within a same single floating point exponent;
generate, for the ML computational operation and based on specifying, to the GPU, the one or more inputs and a second level of precision, a result for the hardware device including one or more hardware intermediate products and a hardware accumulator output using the GPU at the second level of precision; and
output a conformance result based on whether a variance between the reference result and the result is within a threshold range.
10. The computing device of claim 9 , wherein the at least one processor is configured to determine the one or more inputs as in the range of 1.0 and a square root of 2.0.
11. The computing device of claim 9 , wherein the one or more ranges include a first range with an exponent of zero, a second range with a minimum exponent, and a third range with a maximum exponent, and wherein the at least one processor is configured to generate the reference result, generate the result, and compare the reference result to the generated result, for each of the one or more ranges.
12. The computing device of claim 9 , wherein the conformance result includes an indication of whether the generated result passes a conformance test.
13. The computing device of claim 9 , wherein the at least one processor is configured to generate the reference result, generate the result, and compare the reference result to the generated result, for multiple configurations for the computational operation to output conformance results for each of the configurations.
14. The computing device of claim 13 , wherein the multiple configurations include:
a first configuration having a single precision to represent values of the one or more inputs, a single precision to represent values of the one or more hardware intermediate products, a single precision to represent values of an accumulator that generates the hardware accumulator output, and a single precision to represent values of the hardware accumulator output;
a second configuration having a half precision to represent values of the one or more inputs, a half precision to represent values of the one or more hardware intermediate products, a half precision to represent values of an accumulator that generates the hardware accumulator output, and a half precision to represent values of the hardware accumulator output;
a third configuration having a half precision to represent values of the one or more inputs, a half precision to represent values of the one or more hardware intermediate products, a full precision to represent values of an accumulator that generates the hardware accumulator output, and a half precision to represent values of the hardware accumulator output; and
a fourth configuration having a half precision to represent values of the one or more inputs, a full precision to represent values of the one or more hardware intermediate products, a full precision to represent values of an accumulator that generates the hardware accumulator output, and a half precision to represent values of the hardware accumulator output.
15. The computing device of claim 9 , wherein the at least one processor is configured to compare the reference result to the generated result at least in part by comparing each of the one or more reference intermediate products to a corresponding one of the one or more hardware intermediate products and comparing the reference accumulator output to the hardware accumulator output to determine associated error values.
16. The computing device of claim 9 , wherein the ML computational operation includes a convolution operation.
17. A non-transitory computer-readable medium, comprising code executable by one or more processors for performing conformance testing of a machine learning (ML) computational operation configured to be implemented in a graphics processing unit (GPU) wherein the ML computational operation is performed by an algorithm specific to a hardware device, the code comprising code for:
generating, for the ML computational operation and based on one or more inputs, a reference result including one or more reference intermediate products and a reference accumulator output at a first level of precision, wherein the one or more reference intermediate products of the ML computational operation are within one or more ranges including a range where all values within the range are within a same single floating point exponent;
generating, for the ML computational operation and based on specifying, to the GPU, the one or more inputs and a second level of precision, a result for the hardware device including one or more hardware intermediate products and a hardware accumulator output using the GPU at the second level of precision; and
outputting a conformance result based on whether a variance between the reference result and the result is within a threshold range.