Techniques for conformance testing computational operations
Examples described herein generally relate to performing conformance testing of a computational operation. A reference result including one or more reference intermediate products and a reference accumulator output at a first level of precision can be generated for the computational operation and based on one or more inputs. A hardware result can similarly be created using hardware at a second level of precision. The reference result can be compared to the hardware result to determine a variance value. A conformance result can be output based on whether the variance value is within a threshold range.
1. A computer-implemented method for performing conformance testing of a machine learning (ML) computational operation performed by an algorithm specific to a hardware device, comprising:
generating, for the ML computational operation and based on one or more inputs and using a processor and not the hardware device, a reference result including one or more reference intermediate products and a reference accumulator output at a first level of precision, wherein the one or more reference intermediate products of the ML computational operation are within one or more ranges including a range where all values within the range are within a same single floating point exponent;
generating, for the ML computational operation and based on specifying, to the hardware device, the one or more inputs, a result for the hardware device including one or more hardware intermediate products and a hardware accumulator output by the hardware device at a second level of precision;
and
outputting a conformance result based on whether a variance between the reference result and the result is within a threshold range.
2. The computer-implemented method of claim 1 , wherein the one or more ranges include a first range with an exponent of zero, a second range with a minimum exponent, and a third range with a maximum exponent, and wherein generating the reference result, generating the result, and comparing the reference result to the generated result are performed for each of the one or more ranges.
3. The computer-implemented method of claim 1 , wherein the conformance result includes an indication of whether the generated result passes a conformance test.
4. The computer-implemented method of claim 1 , wherein generating the reference result, generating the result, and comparing the reference result to the generated result are performed for multiple configurations for the ML computational operation to output conformance results for each of the multiple configurations.
5. The computer-implemented method of claim 4 , wherein the multiple configurations include:
a first configuration having a single precision to represent values of the one or more inputs, a single precision to represent values of the one or more hardware intermediate products, a single precision to represent values of an accumulator that generates the hardware accumulator output, and a single precision to represent values of the hardware accumulator output;
a second configuration having a half precision to represent values of the one or more inputs, a half precision to represent values of the one or more hardware intermediate products, a half precision to represent values of an accumulator that generates the hardware accumulator output, and a half precision to represent values of the hardware accumulator output;
a third configuration having a half precision to represent values of the one or more inputs, a half precision to represent values of the one or more hardware intermediate products, a full precision to represent values of an accumulator that generates the hardware accumulator output, and a half precision to represent values of the hardware accumulator output; and
a fourth configuration having a half precision to represent values of the one or more inputs, a full precision to represent values of the one or more hardware intermediate products, a full precision to represent values of an accumulator that generates the hardware accumulator output, and a half precision to represent values of the hardware accumulator output.
6. The computer-implemented method of claim 1 , wherein comparing the reference result to the generated result comprises comparing each of the one or more reference intermediate products to a corresponding one of the one or more hardware intermediate products and comparing the reference accumulator output to the hardware accumulator output to determine associated error values.
7. The computer-implemented method of claim 1 , wherein the ML computational operation includes a convolution operation.
8. The computer-implemented method of claim 1 , wherein the one or more inputs are in the range of 1.0 and a square root of 2.0.
9. The computer-implemented method of claim 1 , wherein the hardware device includes one or more processors.
10. A computing device for performing conformance testing of a machine learning (ML) computational operation performed by an algorithm specific to a hardware device, comprising:
at least one memory storing one or more parameters or instructions for developing an application; and
at least one processor coupled to the at least one memory, wherein the at least one processor is configured to:
generate, for the ML computational operation and based on one or more inputs and without using the hardware device, a reference result including one or more reference intermediate products and a reference accumulator output at a first level of precision, wherein the one or more reference intermediate products of the ML computational operation are within one or more ranges including a range where all values within the range are within a same single floating point exponent;
generate, for the ML computational operation and based on specifying, to the hardware device, the one or more inputs, a result for the hardware device including one or more hardware intermediate products and a hardware accumulator output by the hardware device at a second level of precision; and
output a conformance result based on whether a variance between the reference result and the generated result is within a threshold range.
11. The computing device of claim 10 , wherein the one or more ranges include a first range with an exponent of zero, a second range with a minimum exponent, and a third range with a maximum exponent, and wherein the at least one processor is configured to generate the reference result, and generate the result, for each of the one or more ranges.
12. The computing device of claim 10 , wherein the conformance result includes an indication of whether the generated result passes a conformance test.
13. The computing device of claim 10 , wherein the at least one processor is configured to generate the reference result, and generate the result, for multiple configurations for the ML computational operation to output conformance results for each of the multiple configurations.
14. The computing device of claim 13 , wherein the multiple configurations include:
a first configuration having a single precision to represent values of the one or more inputs, a single precision to represent values of the one or more hardware intermediate products, a single precision to represent values of an accumulator that generates the hardware accumulator output, and a single precision to represent values of the hardware accumulator output;
a second configuration having a half precision to represent values of the one or more inputs, a half precision to represent values of the one or more hardware intermediate products, a half precision to represent values of an accumulator that generates the hardware accumulator output, and a half precision to represent values of the hardware accumulator output;
a third configuration having a half precision to represent values of the one or more inputs, a half precision to represent values of the one or more hardware intermediate products, a full precision to represent values of an accumulator that generates the hardware accumulator output, and a half precision to represent values of the hardware accumulator output; and
a fourth configuration having a half precision to represent values of the one or more inputs, a full precision to represent values of the one or more hardware intermediate products, a full precision to represent values of an accumulator that generates the hardware accumulator output, and a half precision to represent values of the hardware accumulator output.
15. The computing device of claim 10 , wherein the at least one processor is configured to output a conformance result based on comparing each of the one or more reference intermediate products to a corresponding one of the one or more hardware intermediate products and comparing the reference accumulator output to the hardware accumulator output to determine associated error values.
16. The computing device of claim 10 , wherein the ML computational operation includes a convolution operation.
17. The computing device of claim 10 , wherein the one or more inputs are in the range of 1.0 and a square root of 2.0.
18. The computing device of claim 10 , wherein the hardware device includes one or more processors.
19. One or more non-transitory computer-readable media, comprising code executable by one or more processors for performing conformance testing of a machine learning (ML) computational operation performed by an algorithm specific to a hardware device, the code comprising code for:
generating, for the ML computational operation and based on one or more inputs and without using the hardware device, a reference result including one or more reference intermediate products and a reference accumulator output at a first level of precision, wherein the one or more reference intermediate products of the ML computational operation are within one or more ranges including a range where all values within the range are within a same single floating point exponent;
generating, for the ML computational operation and based on specifying, to the hardware device, the one or more inputs, a result for the hardware device including one or more hardware intermediate products and a hardware accumulator output by the hardware device at a second level of precision;
and
outputting a conformance result based on whether a variance between the reference result and the result the variance value is within a threshold range.
20. The one or more non-transitory computer-readable media of claim 19 , wherein the one or more ranges include a first range with an exponent of zero, a second range with a minimum exponent, and a third range with a maximum exponent, and wherein the code for generating the reference result generates the reference result for each of the one or more ranges, the code for generating the result generates the result for each of the one or more ranges, and the code for comparing the reference result compares the reference result to the generated result for each of the one or more ranges.