System for optimizing system resources and runtime during a testing procedure
A system for optimizing system resources and runtime during a testing procedure is disclosed. During operation, the system may perform hypothesis testing on an application under evaluation, such as performing hypothesis testing on suggested constraints for the application under evaluation, which may result in the creation of new hypotheses yielding improved confidences. This testing may be optimized by simplifying and/or reducing the number of hypothesis tests used during the testing process. To do so, the hypothesis tests may be sorted into sets, and a subset of these sorted hypothesis tests may be executed. Based on the results of this testing, a remaining portion of the hypothesis tests may be skipped, allowing for an optimization of the testing process without reducing the quality of the testing procedure.
1. A system, comprising: a memory that stores instructions; and a processor that executes the instructions to perform operations, the operations comprising: enumerating a plurality of hypothesis tests; sorting the enumerated hypothesis tests into a plurality of sets; selecting a set of the plurality of sets for testing; determining a subset from the selected set for testing; selecting a test from the subset for testing; executing the test on an application under evaluation, wherein the test has an expected result; obtaining a test result; and determining if testing is ready to be completed.
2. The system of claim 1 , wherein the operations further comprise:
repeating, based on test results, goals, conditions, or a combination thereof, the enumerating, the sorting, the selecting a set, the determining a subset, the selecting a test, the executing, the obtaining the test result, the determining if testing is ready to be completed, or a combination thereof, until testing is ready to be completed; and
completing testing.
3. The system of claim 1 , wherein the enumerating is based, at least in part, on goals.
4. The system of claim 1 , wherein the enumerating a plurality of hypothesis tests comprises receiving the plurality of hypothesis tests, at least in part, from an external source.
5. The system of claim 1 , wherein at least one hypothesis test includes at least one abstract step or abstract expectation, and wherein at least one abstract step or abstract expectation is converted to a concrete step or concrete expectation prior to executing a test determined from the hypothesis test.
6. The system of claim 1 , wherein the sorting, in whole or in part, is performed on an as required basis, the determining a subset, in whole or in part, is performed on an as required basis, or a combination thereof.
7. The system of claim 1 , wherein the selecting a set is performed two or more times, the determining a subset is performed two or more times, or a combination thereof.
8. The system of claim 1 , wherein each of the hypothesis tests are for testing a hypothesis associated with a suggested constraint for the application under evaluation, wherein the suggested constraint is for a field of the application under evaluation, an order of operations of the application under evaluation, a transition of the application under evaluation, or a combination thereof.
9. The system of claim 1 , wherein the operations further comprise:
analyzing the enumerated hypothesis tests to locate at least two hypothesis tests that are simple duplicates of each other in the enumerated hypothesis tests, to locate at least two hypothesis tests that are complex duplicates of each other in the enumerated hypothesis tests, or a combination thereof; and
removing all but one of the hypothesis tests that are simple duplicates of each other from the enumerated hypothesis tests, all but one of the hypothesis tests that are complex duplicates of each other from the enumerated hypothesis tests, or a combination thereof.
10. The system of claim 1 , wherein sorting the enumerated hypothesis tests into a plurality of sets comprises sorting a first portion of the enumerated hypothesis tests into a first set based on a first test feature associated with each of the first portion of the enumerated hypothesis tests; and sorting a second portion of the enumerated hypothesis tests into a second set based on a second test feature associated with each of the second portion of the enumerated hypothesis tests.
11. The system of claim 1 , wherein the operations further comprise training or improving a model, parameter, weight, dictionary, threshold, confidence, or filter associated with generating a future hypothesis; hypothesis test, test set, or test subset; hypothesis test, test set, or test subset exclusion; hypothesis test, test set, or test subset optimization; hypothesis test, test set, or test subset ordering; hypothesis test, test set, or test subset selection; or hypothesis test, test set, or test subset enumeration; wherein the training or improving is based on feedback, hypothesis test results, an internal document, an external document, the application under evaluation, the hypothesis, a similar hypothesis, a related hypothesis, a hypothesis test, a hypothesis abstract test, a hypothesis concrete test, a hypothesis test set, a hypothesis test subset, a hypothesis testing goal, a hypothesis testing enumeration, a hypothesis strategy, a hypothesis test ordering or classification, or any other source or combination thereof.
12. A method, comprising: enumerating, by utilizing instructions from a memory that are executed by a processor, a plurality of hypothesis tests; sorting the enumerated hypothesis tests into a plurality of sets; selecting a set of the plurality of sets for testing; determining a subset from the selected set for testing; selecting a test from the subset for testing; executing the test on an application under evaluation, wherein the test has an expected result; obtaining the test result; repeating, based on test results, goals, conditions, or a combination thereof, the enumerating, the sorting, the selecting a set, the determining a subset, the selecting a test, the executing, the obtaining a test result, or a combination thereof, until testing is ready to be completed; and completing testing.
13. The method of claim 12 , wherein determining a subset from the selected set for testing comprises ordering the tests in the selected set based on test quality; and selecting only the ordered tests that have a respective test quality greater than a threshold.
14. The method of claim 12 , wherein determining a subset from the selected set for testing comprises ordering the tests in the selected set based on test runtime; and selecting only the ordered tests that have a respective test runtime less than a threshold.
15. The method of claim 12 , wherein determining a subset from the selected set for testing comprises ordering the tests in the selected set based on test cost; and selecting only the ordered tests that have a respective test cost less than a threshold.
16. The method of claim 12 , wherein determining a subset from the selected set for testing comprises ordering the tests in the selected set based on confidence level; and selecting only the ordered tests that have a respective confidence level greater than a threshold.
17. The method of claim 12 , wherein determining a subset from the selected set for testing comprises selecting only the tests in the selected set that have a positive expected result, or selecting only the tests in the selected set that have a negative expected result.
18. The method of claim 12 , wherein determining a subset from the selected set for testing comprises analyzing the tests included in the selected set to determine a portion of the tests that test a field that has a historically positive result; and selecting one or more of the tests included in the determined portion of the tests.
19. The method of claim 12 , wherein determining a subset from the selected set for testing comprises analyzing the tests included in the selected set to determine a portion of the tests that test a field that has a historically negative result; and selecting one or more of the tests included in the determined portion of the tests.
20. A non-transitory computer-readable device comprising instructions, which when loaded and executed by a processor, cause the processor to perform operations comprising: enumerating a plurality of hypothesis tests based on one or more goals; sorting the enumerated hypothesis tests into a plurality of sets; selecting a set of the plurality of sets for testing; determining a subset from the selected set for testing; selecting a test from the subset for testing; executing the test on an application under evaluation; obtaining the test result; determining if testing is ready to be completed; following a determination that the testing is not ready to be completed, repeating the enumerating, the sorting, the selecting a set, the determining a subset, the selecting a test, the executing, the obtaining a test result, the determining if testing is ready to be completed, or a combination thereof, until testing is ready to be completed; and completing testing.
21. The non-transitory computer-readable device of claim 20 , wherein at least one hypothesis test and one executed test are the same.
22. The non-transitory computer-readable device of claim 20 , wherein determining a subset from the selected set for testing comprises determining a historical test value for each of the tests included in the selected set; and selecting only the tests included in the selected set that have a respective historical test value that is greater than a threshold.