Validation of a system using a design of experiments processing technique
A validation system includes a test block that operates to apply a set of inputs to a system under test, such as a test system or an executable test algorithm, and receive from said system under test a first set of outputs produced by operation of the system under test in response to application of the set of inputs. The first set of outputs, as well as a second set of outputs reflecting output produced by operation of a reference system or executable reference algorithm in response to application of the same set of inputs, is processed to make a validation determination. A validation processing block compares the first and second sets of outputs to validate the system under test as an equivalent to the reference system.
1. A system, comprising:
software components including a reference algorithm;
hardware components including a processing component operable to execute the reference algorithm; and
a validation system configured to test execution of a test algorithm in comparison to execution of said reference algorithm by: applying a set of inputs selected in accordance with a Design of Experiments (DoE) methodology to execution by both the reference algorithm and test algorithm to produce a first set of outputs and a second set of outputs, respectively, and in response thereto validate the test algorithm to replace said reference algorithm for execution by said processing component by: tabulating means and variances of the first and second sets of outputs and generating a table of DoE effects from the tabulated means and variances as intervals to a desired level of confidence concerning equivalence in execution of the test algorithm and reference algorithm.
2. The system of claim 1 , wherein said system comprises an integrated circuit System on Chip (SoC).
3. The system of claim 1 , wherein said desired level of confidence concerning equivalence is indicative of whether the test algorithm is an acceptable replacement for the reference algorithm.
4. The system of claim 1 , wherein said desired level of confidence concerning equivalence is indicative of whether the reference algorithm or test algorithm is better performing.
5. The system of claim 1 , wherein said desired level of confidence concerning equivalence is indicative of whether the reference algorithm and test algorithm are functionally different.
6. The system of claim 1 , wherein validation comprises determining validation of the test algorithm based on a confidence interval.
7. A system, comprising:
software components including a test algorithm;
hardware components including a processing component operable to execute the test algorithm; and
a built-in-self-test system configured to test execution of the test algorithm in comparison to execution of a reference algorithm by: applying a set of inputs selected in accordance with a Design of Experiments (DoE) methodology to execution by both the reference algorithm and test algorithm to produce a first set of outputs and a second set of outputs, respectively, and in response thereto validate that execution of test algorithm by said processing component conforms with execution of said reference algorithm by: tabulating means and variances of the first and second sets of outputs and generating a table of DoE effects from the tabulated means and variances as intervals to a desired level of confidence concerning equivalence in execution of the test algorithm and reference algorithm.
8. The system of claim 7 , wherein said system comprises an integrated circuit System on Chip (SoC).
9. The system of claim 7 , wherein said desired level of confidence concerning equivalence indicates whether the test algorithm performs in accordance with the reference algorithm.
10. The system of 9 , wherein the validation system is further operable to generate an output in response to said validation enabling operation of the system with the test algorithm.
11. The system of claim 7 , wherein validation comprises determining validation of the test algorithm based on a confidence interval.
12. A method, comprising:
accessing a reference algorithm;
executing the accessed reference algorithm in response to a set of inputs selected in accordance with a Design of Experiments (DoE) methodology to produce a first set of outputs;
accessing a test algorithm;
executing the accessed test algorithm in response to the same set of inputs to produce a second set of outputs; and
comparing the first and second sets of outputs to validate the test algorithm as an equivalent to the reference algorithm by:
tabulating means and variances of the first and second sets of outputs; and
generating a table of DoE effects from the tabulated means and variances as intervals to a desired level of confidence concerning equivalence in execution of the test algorithm and reference algorithm.
13. The method of claim 12 , further comprising replacing the reference algorithm with the test algorithm for execution in a processing system in response to said comparison.
14. The method of claim 12 , further comprising enabling operation of a processing system executing the test algorithm in response to said comparison.
15. The method of claim 12 , wherein comparing further comprises: determining validation of the test algorithm based on a confidence interval.
16. The method of claim 12 , wherein a result of said comparison provides an indication of whether the reference algorithm or test algorithm is better performing.
17. The method of claim 12 , wherein a result of said comparison provides an indication of whether the reference algorithm and test algorithm are functionally different.
18. A validation system, comprising:
a test block configured to apply a set of inputs selected in accordance with a Design of Experiments (DoE) methodology to a system under test and receive from said system under test a first set of outputs produced by operation of the system under test in response to application of the set of inputs;
a memory storing the first set of outputs and further storing a second set of outputs, said second set of outputs reflecting output produced by operation of a reference system in response to application of the same set of inputs; and
a validation processing block configured to compare the first and second sets of outputs to validate the system under test as an equivalent to the reference system by:
tabulating means and variances of the first and second sets of outputs; and
generating a table of DoE effects from the tabulated means and variances as intervals to a desired level of confidence concerning equivalence in execution of the test algorithm and reference algorithm.
19. The validation system of claim 18 , wherein the validation system is a built-in-self-test (BIST) system.
20. The validation system of claim 18 , wherein the system under test comprises a test algorithm and operation the system under test comprises execution of the test algorithm in response to the set of inputs, and further wherein the reference system comprises a reference algorithm, said second set of outputs produced from execution of the reference algorithm in response to the set of inputs.
21. The validation system of claim 18 , wherein the validation system is a component of a System on Chip (SoC).
22. The validation system of claim 18 , wherein said validation processing block is further configured to determine validation of the system under test based on a confidence interval.
23. A method, comprising:
applying a set of inputs selected in accordance with a Design of Experiments (DoE) methodology to a system under test;
receiving from said system under test a first set of outputs produced by operation of the system under test in response to application of the set of inputs;
storing the first set of outputs;
storing a second set of outputs, said second set of outputs reflecting output produced by operation of a reference system in response to application of the same set of inputs; and
comparing the first and second sets of outputs to validate the system under test as an equivalent to the reference system by:
tabulating means and variances of the first and second sets of outputs; and
generating a table of DoE effects from the tabulated means and variances as intervals to a desired level of confidence concerning equivalence between the system under test and reference system.
24. The method of claim 23 , wherein the system under test comprises a test algorithm and operation the system under test comprises executing the test algorithm in response to the set of inputs, and further wherein the reference system comprises a reference algorithm, said second set of outputs produced from execution of the reference algorithm in response to the set of inputs.
25. The method of claim 23 , wherein the method further comprising: determining validation of the system under test based on a confidence interval.