TEST CASE EXECUTION SEQUENCES
A system and method reorder execution of a test suite to be performed on a given device according to an initial testing order. Each testing sequence in the test suite is analyzed for dependencies between test cases, and these dependencies are recorded in directed graphs. Next, a machine learning algorithm, such as the random forest algorithm, is trained on multi-dimensional historical testing data according to several testing parameters to predict success or failure of any given test. The trained algorithm is used to predict, for a given device under test, which of the test cases are likely to fail, and to compute a confidence value for each such prediction. The directed graphs then are reorganized so that graphs containing tests most likely to fail are executed early in the test suite, according to a modified testing order that accounts for both test dependencies and the confidence values.
1 . A system for reordering execution of a test suite that is stored in a test suite database and that comprises a plurality of tests to be performed on a given device according to an initial order, the system comprising:
a graph processor for creating a plurality of directed graphs comprising nodes and edges, wherein each node represents a test in the plurality of tests and each edge from a first node to a second node represents creation of an output, by the first node, that is used as an input by the second node;
a training database for storing parametric training data obtained from performance of the test suite on devices other than the given device;
a prediction processor for using a machine learning algorithm, trained using data stored in the training database, to:
predict, for each test in the plurality of tests, whether performance of that test on the given device is likelier to succeed or fail according to parametric data for the given device, and
generate a confidence value for each such prediction; and
a reordering processor for creating, for performance on the given device, a test suite comprising the plurality of tests rearranged according to a modified order;
wherein at least one test, predicted to fail by the prediction processor, appears earlier in the modified order than in the initial order.
2 . A system according to claim 1 , wherein the training data comprise a plurality of records, each record relating to a test and including data indicating both success or failure of the test, and one or more of: a unique device identifier, a device operating system identifier, a device testing application version, a device model identifier, a test identifier, a test cycle number, a dependency tree identifier, and a dependency tree level identifier.
3 . A system according to claim 1 , wherein the prediction processor is configured to use a random forest machine learning algorithm to:
predict performance of at least one test in the plurality of tests by aggregating predictions of a plurality of decision trees in a random forest; and
generate the confidence value as a ratio of (a) the number of decision trees within the plurality of decision trees whose predictions agree with the predicted performance, to (b) the number of trees in the plurality of trees.
4 . A system according to claim 1 , wherein the reordering processor creates the test suite according to the modified order by:
determining a set of directed paths, in the plurality of directed graphs, that each end on a node that represents a test that was predicted likelier to fail than successful;
ordering the set of directed graphs by increasing length of the shortest directed path therein; and
further ordering the set of directed graphs by decreasing maximum confidence value.
5 . A system according to claim 4 , wherein determining the set of directed paths includes, for each test that was predicted likelier to fail than successful, identifying the edges in a corresponding directed path for that test by traversing the directed graph that comprises the test from the node representing that test to a root node.
6 . A system according to claim 1 , further comprising a plurality of testing processors, each testing processor in the plurality configured to perform, on the given device, the tests represented by nodes in a directed path according to the modified order.
7 . A method of reordering execution of a test suite that is stored in a test suite database and that comprises a plurality of tests to be performed on a given device according to an initial order, the method comprising:
creating a plurality of directed graphs comprising nodes and edges, wherein each node represents a test in the plurality of tests and each edge from a first node to a second node represents creation of an output, by the first node, that is used as an input by the second node;
storing, in a training database, parametric training data obtained from performance of the test suite on devices other than the given device;
using a machine learning algorithm, trained using the stored parametric training data, to:
predict, for each test in the plurality of tests, whether performance of that test on the given device is likelier to succeed or fail according to parametric data for the given device, and
generate a confidence value for each such prediction; and
creating, for performance on the given device, a test suite comprising the plurality of tests rearranged according to a modified order;
wherein at least one test, predicted to fail by the prediction processor, appears earlier in the modified order than in the initial order.
8 . A method according to claim 7 , wherein the training data comprise a plurality of records, each record relating to a test and including data indicating both success or failure of the test, and one or more of: a unique device identifier, a device operating system identifier, a device testing application version, a device model identifier, a test identifier, a test cycle number, a dependency tree identifier, and a dependency tree level identifier.
9 . A method according to claim 7 , wherein predicting performance of at least one test in the plurality of tests comprises aggregating predictions of a plurality of decision trees in a random forest, and wherein generating the confidence value comprises computing a ratio of (a) the number of decision trees within the plurality of decision trees whose predictions agree with the predicted performance, to (b) the number of trees in the plurality of trees.
10 . A method according to claim 7 , wherein creating the test suite according to the modified order comprises:
determining a set of directed paths, in the plurality of directed graphs, that each end on a node that represents a test that was predicted likelier to fail than successful;
ordering the set of directed graphs by increasing length of the shortest directed path therein; and
further ordering the set of directed graphs by decreasing maximum confidence value.
11 . A method according to claim 10 , wherein determining the set of directed paths includes, for each test that was predicted likelier to fail than successful, identifying the edges in a corresponding directed path for that test by traversing the directed graph that comprises the test from the node representing that test to a root node.
12 . A method according to claim 7 , further comprising performing, on the given device by each of a plurality of testing processors, the tests represented by nodes in a corresponding directed path according to the modified order.
13 . A method according to claim 12 , further comprising:
storing, in the training database, parametric training data obtained from performing the tests according to the modified order; and
retraining the machine learning algorithm using the updated, stored parametric training data.
14 . A tangible, computer-readable storage medium, in which is non-transitorily stored computer program code for performing a method of reordering execution of a test suite that is stored in a test suite database and that comprises a plurality of tests to be performed on a given device according to an initial order, the method comprising:
creating a plurality of directed graphs comprising nodes and edges, wherein each node represents a test in the plurality of tests and each edge from a first node to a second node represents creation of an output, by the first node, that is used as an input by the second node;
storing, in a training database, parametric training data obtained from performance of the test suite on devices other than the given device;
using a machine learning algorithm, trained using the stored parametric training data, to:
predict, for each test in the plurality of tests, whether performance of that test on the given device is likelier to succeed or fail according to parametric data for the given device, and
generate a confidence value for each such prediction; and
creating, for performance on the given device, a test suite comprising the plurality of tests rearranged according to a modified order;
wherein at least one test, predicted to fail by the prediction processor, appears earlier in the modified order than in the initial order.
15 . A storage medium according to claim 14 , wherein the training data comprise a plurality of records, each record relating to a test and including data indicating both success or failure of the test, and one or more of: a unique device identifier, a device operating system identifier, a device testing application version, a device model identifier, a test identifier, a test cycle number, a dependency tree identifier, and a dependency tree level identifier.
16 . A storage medium according to claim 14 , wherein predicting performance of at least one test in the plurality of tests comprises aggregating predictions of a plurality of decision trees in a random forest, and wherein generating the confidence value comprises computing a ratio of (a) the number of decision trees within the plurality of decision trees whose predictions agree with the predicted performance, to (b) the number of trees in the plurality of trees.
17 . A storage medium according to claim 14 , wherein creating the test suite according to the modified order comprises:
determining a set of directed paths, in the plurality of directed graphs, that each end on a node that represents a test that was predicted likelier to fail than successful;
ordering the set of directed graphs by increasing length of the shortest directed path therein; and
further ordering the set of directed graphs by decreasing maximum confidence value.
18 . A storage medium according to claim 17 , wherein determining the set of directed paths includes, for each test that was predicted likelier to fail than successful, identifying the edges in a corresponding directed path for that test by traversing the directed graph that comprises the test from the node representing that test to a root node.
19 . A storage medium according to claim 14 , wherein the method further comprises performing, on the given device by one or more testing processors, the tests represented by nodes in a directed path according to the modified order.
20 . A storage medium according to claim 19 , wherein the method further comprises:
storing, in the training database, parametric training data obtained from performance of the test suite on the given device; and
retraining the machine learning algorithm using the updated, stored parametric training data.