REDUCING TIME TO TEST CYCLE FIRST FAIL
A system that automatically reduces test cycle time to save resources and developer time. The present system selects a subset of tests from a full test plan that should be selected for a particular test cycle, rather than running the entire test plan. The subset of tests is intelligently selected using metrics such as tests associated with changed code and new and modified tests.
1 . A method for automatically testing software code, comprising:
detecting a test event initiated by a testing program and associated with testing a first software at a testing server, the test event detected by an agent executing within the testing program at the testing server, the testing event associated with a plurality of tests for the first software;
receiving, by the agent on the testing server from the remote server, a list of tests to be performed in response to the test event, the received list of tests being a subset of the plurality of tests;
ordering each test in the list of tests according to a likelihood of failure; and
executing the ordered tests by the agent in the testing server.
2 . The method of claim 1 , further comprising generating, by a score generator, a likelihood of failure for each test, wherein the ordering is based on the likelihood of failure for each test.
3 . The method of claim 1 , wherein the score generator predicts the likelihood of failure based on historical data.
4 . The method of claim 1 , wherein the score generator predicts the likelihood of failure based on source code change data.
5 . The method of claim 4 , wherein the source code change data includes a source code change time, files that were added or modified to the source code, level of change made to the software, and who made the change to the source code.
6 . The method of claim 1 , wherein ordering each test includes:
training a score generator using training data associated with the source code being tested; and
applying test data to the trained score generator to generate the likelihood of failure for each test in the test list.
7 . The method of claim 1 , wherein the list of tests is generated based on a first call graph having relationships between the plurality of tests and portions of the first software, the duration of execution of the subset of tests in the test list is shorter than the duration of execution of the plurality of tests.
8 . A non-transitory computer readable storage medium having embodied thereon a program, the program being executable by a processor to perform a method for automatically testing software code, the method comprising:
detecting a test event initiated by a testing program and associated with testing a first software at a testing server, the test event detected by an agent executing within the testing program at the testing server, the testing event associated with a plurality of tests for the first software;
receiving, by the agent on the testing server from the remote server, a list of tests to be performed in response to the test event, the received list of tests being a subset of the plurality of tests;
ordering each test in the list of tests according to a likelihood of failure; and
executing the ordered tests by the agent in the testing server.
9 . The non-transitory computer readable storage medium of claim 8 , further comprising generating, by a score generator, a likelihood of failure for each test, wherein the ordering is based on the likelihood of failure for each test.
10 . The non-transitory computer readable storage medium of claim 8 , wherein the score generator predicts the likelihood of failure based on historical data.
11 . The non-transitory computer readable storage medium of claim 8 , wherein the score generator predicts the likelihood of failure based on source code change data.
12 . The non-transitory computer readable storage medium of claim 11 , wherein the source code change data includes a source code change time, files that were added or modified to the source code, level of change made to the software, and who made the change to the source code.
13 . The non-transitory computer readable storage medium of claim 8 , wherein ordering each test includes:
training a score generator using training data associated with the source code being tested; and
applying test data to the trained score generator to generate the likelihood of failure for each test in the test list.
14 . The non-transitory computer readable storage medium of claim 8 , wherein the list of tests is generated based on a first call graph having relationships between the plurality of tests and portions of the first software, the duration of execution of the subset of tests in the test list is shorter than the duration of execution of the plurality of tests.
15 . A system for automatically testing software code, comprising:
a server including a memory and a processor; and
one or more modules stored in the memory and executed by the processor to detect a test event initiated by a testing program and associated with testing a first software at a testing server, the test event detected by an agent executing within the testing program at the testing server, the testing event associated with a plurality of tests for the first software, receive, by the agent on the testing server from the remote server, a list of tests to be performed in response to the test event, the received list of tests being a subset of the plurality of tests, order each test in the list of tests according to a likelihood of failure, and execute the ordered tests by the agent in the testing server.
16 . The system of claim 15 , modules further executable to generate, by a score generator, a likelihood of failure for each test, wherein the ordering is based on the likelihood of failure for each test.
17 . The system of claim 15 , wherein the score generator predicts the likelihood of failure based on historical data.
18 . The system of claim 15 , wherein the score generator predicts the likelihood of failure based on source code change data.
19 . The system of claim 18 , wherein the source code change data includes a source code change time, files that were added or modified to the source code, level of change made to the software, and who made the change to the source code.
20 . The method of claim 15 , wherein ordering each test includes:
training a score generator using training data associated with the source code being tested; and
applying test data to the trained score generator to generate the likelihood of failure for each test in the test list.