Variability system and analytics for continuous reliability in cloud-based workflows
The present disclosure relates to automatically detecting software defects and determining the source of the defect. An example method generally includes receiving an output data set from a host server executing a test operation and comparing the output data set to a performance model. A test system identifies that a defect exists in software executing on the host server based on determining that the output data set deviates from the performance model by more than a threshold. The test system retrieves a source code update from a source code repository that was committed to the source code repository before the execution of the test operation. The test system compares a scope of the source code update to the defect and notifies a development team related to the source code update of the defect upon determining that the scope of the source code update and the defect match.
1. A method for detecting and sourcing software defects comprising:
receiving an output data set from a host server executing a test operation in a software system;
comparing the output data set to a performance model;
identifying that a defect exists in software executing on the host server based, at least in part, on determining that the output data set deviates from the performance model by more than a threshold;
retrieving a source code update from a source code repository, wherein the source code update was committed to the source code repository before the execution of the test operation;
comparing a scope of the source code update to the defect; and
notifying a development resource related to the source code update of the defect upon determining that the scope of the source code update and the defect match.
2. The method of claim 1 , further comprising:
generating a visualization including the performance model and the output data set; and
displaying the visualization to a user.
3. The method of claim 2 , wherein generating the visualization comprises generating performance data graphs for a plurality of subsystems of the software system, wherein each performance data graph illustrates the performance model and the output data set for a specific subsystem of the plurality of subsystems.
4. The method of claim 1 , wherein the performance model is based, at least in part, on performance data captured in a plurality of prior output data sets.
5. The method of claim 1 , wherein the output data comprises a score weighing performance of each of a plurality of subsystems of the software system, and wherein identifying that the defect exists comprises determining that the score differs from a threshold score defined by the performance model by a threshold amount.
6. The method of claim 1 , further comprising:
identifying a second source code update deployed prior to a commit date of the source code update; and
deploying the second source code update to the host server.
7. The method of claim 1 , wherein the output data set includes metadata describing a software version that the test operation was executed in, metadata defining operations performed as part of the test operation, and metadata recording a response time for each operation.
8. A system, comprising:
a processor; and
memory storing instructions which, when executed on the processor, perform an operation for detecting and sourcing software defects, the operation comprising:
receiving an output data set from a host server executing a test operation in a software system;
comparing the output data set to a performance model;
identifying that a defect exists in software executing on the host server based, at least in part, on determining that the output data set deviates from the performance model by more than a threshold;
retrieving a source code update from a source code repository, wherein the source code update was committed to the source code repository before the execution of the test operation;
comparing a scope of the source code update to the defect; and
notifying a development team related to the source code update of the defect upon determining that the scope of the source code update and the defect match.
9. The system of claim 8 , the operation further comprising:
generating a visualization including the performance model and the output data set; and
displaying the visualization to a user.
10. The system of claim 9 , wherein generating the visualization comprises generating performance data graphs for a plurality of subsystems of the software system, wherein each performance data graph illustrates the performance model and the output data set for a specific subsystem of the plurality of subsystems.
11. The system of claim 8 , wherein the performance model is based, at least in part, on performance data captured in a plurality of prior output data sets.
12. The system of claim 8 , wherein the output data comprises a score weighing performance of each of a plurality of subsystems of the software system, and wherein identifying that the defect exists comprises determining that the score differs from a threshold score defined by the performance model by a threshold amount.
13. The system of claim 8 , further comprising:
identifying a second source code update deployed prior to a commit date of the source code update; and
deploying the second source code update to the host server.
14. The system of claim 8 , wherein the output data set includes metadata describing a software version that the test operation was executed in, metadata defining operations performed as part of the test operation, and metadata recording a response time for each operation.
15. A non-transitory computer-readable medium comprising instructions which, when executed by one or more processors, performs an operation for detecting and sourcing software defects, the operation comprising:
receiving an output data set from a host server executing a test operation in a software system;
comparing the output data set to a performance model;
identifying that a defect exists in software executing on the host server based, at least in part, on determining that the output data set deviates from the performance model by more than a threshold;
retrieving a source code update from a source code repository, wherein the source code update was committed to the source code repository before the execution of the test operation;
comparing a scope of the source code update to the defect; and
notifying a development team related to the source code update of the defect upon determining that the scope of the source code update and the defect match.
16. The non-transitory computer-readable medium of claim 15 , wherein the operation further comprises:
generating a visualization including the performance model and the output data set; and
displaying the visualization to a user.
17. The non-transitory computer-readable medium of claim 16 , wherein generating the visualization comprises generating performance data graphs for a plurality of subsystems of the software system, wherein each performance data graph illustrates the performance model and the output data set for a specific subsystem of the plurality of subsystems.
18. The non-transitory computer-readable medium of claim 15 , wherein the performance model is based, at least in part, on performance data captured in a plurality of prior output data sets.
19. The non-transitory computer-readable medium of claim 15 , the output data comprises a score weighing performance of each of a plurality of subsystems of the software system, and wherein identifying that the defect exists comprises determining that the score differs from a threshold score defined by the performance model by a threshold amount.
20. The non-transitory computer-readable medium of claim 15 further comprising:
identifying a second source code update deployed prior to a commit date of the source code update; and
deploying the second source code update to the host server.