Forecasting a quality of a software release using machine learning
In some examples, a server may retrieve and parse test results associated with testing a software package. The server may determine a weighted sum of a software feature index associated with a quality of the plurality of features, a defect index associated with the defects identified by the test cases, a test coverage index indicating a pass rate of the plurality of test cases, a release release reliability index associated with results of executing regression test cases included in the test cases, and an operational quality index associated with resources and an environment associated with the software package. The server may use a machine learning algorithm, such as a time series forecasting algorithm, to forecast a release status of the software package. The server may determine, based on the release status, whether the software package is to progress from a current phase to a next phase of a development cycle.
1. A method comprising:
retrieving, by one or more processors, test results associated with a software package comprising a plurality of features, the test results created by executing a plurality of test cases to test the plurality of features;
parsing, by the one or more processors, the test results to create parsed data;
determining, by the one or more processors and based on the parsed data, a software feature index associated with a quality of the plurality of features at a particular point in time of a development cycle;
determining, by the one or more processors and based on the parsed data, a defect index associated with the defects identified by the test cases;
determining, by the one or more processors and based on the parsed data, a test coverage index indicating a pass rate of the plurality of test cases;
determining, by the one or more processors and based on the parsed data, a release reliability index associated with results of executing regression test cases included in the test cases;
determining, by the one or more processors and based on the parsed data, an operational quality index associated with resources and an environment associated with the software package;
generating, by the one or more processors, a time series of release indexes, wherein the release index at a given time within the time series is determined based at least in part on the software feature index, the defect index, the test coverage index, the release reliability index, and the operational quality index occurring at the given time in the time series;
determining, by the one or more processors, whether the release indexes in the time series are stationary;
applying, by the one or more processors, the time series of release indexes to a sequence of machine learning models to identify a machine learning model in which residuals appear as white noise if it is determined that the time series of release indexes is stationary;
forecasting, by the one or more processors, a release status of the software package using the identified machine learning model, wherein the identified machine learning model employs release index values of the time series of release indexes from more than one phase of the development cycle to forecast the release status of the software package; and
modifying, by the one or more processors, one or more features of the plurality of features of the software package upon determining that a subsequent release status of the software package in a subsequent phase of the development cycle fails to satisfy a predetermined threshold, wherein the subsequent release status is forecast using a subsequent machine learning model employing one or more release index values among the previous release index values and a subsequent release index value determined based at least in part on a subsequent software feature index, a subsequent defect index, a subsequent test coverage index, a subsequent release reliability index, and a subsequent operational quality index of the software package determined in the subsequent phase of the development cycle.
2. The method of claim 1 , further comprising:
applying, by the one or more processors, a difference algorithm to the time series of release indexes in response to a determination that the time series of release indexes is not stationary, wherein application of the difference algorithm renders the time series of release indexes stationary.
3. The method of claim 1 , wherein determining the release index comprises determining a sum of:
multiplying a first weight with the software feature index,
multiplying a second weight with the defect index,
multiplying a third weight with the test coverage index,
multiplying a fourth weight with the release reliability index, and
multiplying a fifth weight with the operational quality index.
4. The method of claim 1 , wherein the sequence of machine learning models comprises one or more auto regressive integrated moving average (ARIMA) time series forecasting models.
5. The method of claim 4 , further comprising:
applying, by the one or more processors, an auto-correlation operation to the time series of release indexes to select order parameters for the one or more ARIMA time series forecasting models.
6. The method of claim 1 , wherein the software feature index is determined based at least in part on:
an implementation accuracy associated with the plurality of features; and
a number of unimplemented features in the software package.
7. The method of claim 1 , wherein the defect index is determined based at least in part on:
a number of current defects;
a severity of the current defects;
a priority of the current defects; and
a current phase in the development cycle in which the current defects were determined.
8. A computing device comprising:
one or more processors; and
one or more non-transitory computer readable media storing instructions executable by the one or more processors to perform operations comprising:
retrieving test results associated with a software package comprising a plurality of features, the test results created by executing a plurality of test cases to test the plurality of features;
parsing the test results to create parsed data;
determining, based on the parsed data, a software feature index associated with a quality of the plurality of features;
determining, based on the parsed data, a defect index associated with the defects identified by the test cases;
determining, based on the parsed data, a test coverage index indicating a pass rate of the plurality of test cases;
determining, based on the parsed data, a release reliability index associated with results of executing regression test cases included in the test cases;
determining, based on the parsed data, an operational quality index associated with resources and an environment associated with the software package;
generating, by the one or more processors, a time series of release indexes, wherein the release index at a given time within the times series is determined based at least in part on the software feature index, the defect index, the test coverage index, the release reliability index, and the operational quality index occurring at the given time in the time series;
determining, by the one or more processors, whether the release indexes in the time series are stationary;
applying, by the one or more processors, the time series of release indexes to a sequence of machine learning models to identify a machine learning model in which residuals appear as white noise if it is determined that the time series of release indexes is stationary;
forecasting, by the one or more processors, a release status of the software package using the identified machine learning model, wherein the identified machine learning model employs release index values of the time series of release indexes from more than one phase of the development cycle to forecast the release status of the software package; and
modifying, by the one or more processors, one or more features of the plurality of features of the software package upon determining that a subsequent release status of the software package in a subsequent phase of the development cycle fails to satisfy a predetermined threshold, wherein the subsequent release status is forecast using a subsequent machine learning model employing one or more release index values among the previous release index values and a subsequent release index value determined based at least in part on a subsequent software feature index, a subsequent defect index, a subsequent test coverage index, a subsequent release reliability index, and a subsequent operational quality index of the software package determined in the subsequent phase of the development cycle.
9. The computing device of claim 8 , further comprising:
applying, by the one or more processors, a difference algorithm to the time series of release indexes in response to a determination that the time series of release indexes is not stationary, wherein application of the difference algorithm renders the time series of release indexes stationary.
10. The computing device of claim 8 , wherein determining the release index comprises determining a sum of:
multiplying a first weight with the software feature index,
multiplying a second weight with the defect index,
multiplying a third weight with the test coverage index,
multiplying a fourth weight with the release reliability index, and
multiplying a fifth weight with the operational quality index.
11. The computing device of claim 8 , wherein the sequence of machine learning models comprises one or more auto regressive integrated moving average (ARIMA) time series forecasting models.
12. The computing device of claim 11 , further comprising:
applying an auto-correlation operation to the time series of release indexes to select order parameters for the one or more ARIMA time series forecasting models.
13. The computing device of claim 8 , wherein the defect index is determined based at least in part on:
a number of current defects;
a severity of the current defects;
a priority of the current defects; and
a current phase in a development cycle in which the current defects were determined.
14. One or more non-transitory computer readable media storing instructions executable by one or more processors to perform operations comprising:
retrieving test results associated with a software package comprising a plurality of features, the test results created by executing a plurality of test cases to test the plurality of features;
parsing the test results to create parsed data;
determining, based on the parsed data, a software feature index associated with a quality of the plurality of features;
determining, based on the parsed data, a defect index associated with the defects identified by the test cases;
determining, based on the parsed data, a test coverage index indicating a pass rate of the plurality of test cases;
determining, based on the parsed data, a release reliability index associated with results of executing regression test cases included in the test cases;
determining, based on the parsed data, an operational quality index associated with resources and an environment associated with the software package;
generating, by the one or more processors, a time series of release indexes, wherein the release index at a given time within the time series is determined based at least in part on the software feature index, the defect index, the test coverage index, the release reliability index, and the operational quality index occurring at the given time in the time;
determining, by the one or more processors, whether the release indexes in the time series are stationary;
applying, by the one or more processors, the time series of release indexes to a sequence of machine learning models to identify a machine learning model in which residuals appear as white noise if it is determined that the time series of release indexes is stationary;
forecasting, by the one or more processors, a release status of the software package using the identified machine learning model, wherein the identified machine learning model employs release index values of the time series of release indexes from more than one phase of the development cycle to forecast the release status of the software package; and
modifying, by the one or more processors, one or more features of the plurality of features of the software package upon determining that a subsequent release status of the software package in a subsequent phase of the development cycle fails to satisfy a predetermined threshold, wherein the subsequent release status is forecast using a subsequent machine learning model employing one or more release index values among the previous release index values and a subsequent release index value determined based at least in part on a subsequent software feature index, a subsequent defect index, a subsequent test coverage index, a subsequent release reliability index, and a subsequent operational quality index of the software package determined in the subsequent phase of the development cycle.
15. The one or more non-transitory computer readable media of claim 14 , wherein the operations further comprise:
applying, by the one or more processors, a difference algorithm to the time series of release indexes in response to a determination that the time series of release indexes is not stationary, wherein application of the difference algorithm renders the time series of release indexes stationary.
16. The one or more non-transitory computer readable media of claim 14 , wherein determining the release index comprises determining a sum of:
multiplying a first weight with the software feature index,
multiplying a second weight with the defect index,
multiplying a third weight with the test coverage index,
multiplying a fourth weight with the release reliability index, and
multiplying a fifth weight with the operational quality index.
17. The one or more non-transitory computer readable media of claim 14 , wherein the sequence of machine learning models comprises one or more auto regressive integrated moving average (ARIMA) time series forecasting models.
18. The one or more non-transitory computer readable media of claim 17 , further comprising:
applying, by the one or more processors, an auto-correlation operation to the time series of release indexes to select order parameters for the one or more ARIMA time series forecasting models.
19. The one or more non-transitory computer readable media of claim 18 , further comprising:
applying, by the one or more processors, a logarithmic operation to the time-series of release indexes.
20. The one or more non-transitory computer readable media of claim 14 , wherein the defect index is determined based at least in part on:
a number of current defects;
a severity of the current defects;
a priority of the current defects; and
a current phase in the development cycle in which the current defects were determined.