Automated bug detection with virtual machine forking
Techniques for automated bug detection. A set of inputs are collected and a snapshotting feature is used to apply each input to a test application. Outputs from the test application are gathered and compared to determine whether the outputs are associated with bugs. Comparison can be done with one or more of many different techniques that quantify difference between outputs associated with test inputs and outputs associated with a “happy path input.” Outputs can be grouped together based on these quantifications and the groups can be used to identify outputs most likely to be associated with bugs. The output groups may also be used to group associated inputs to the set of inputs to be used for testing in the future. When a bug is identified, a report could be automatically generated that includes a scoring value as well as recorded output information and could be presented to a user.
1. A method for detecting bugs in an application, the method comprising:
executing a test application in a first virtual machine;
creating a snapshot associated with the first virtual machine and the test application at a fork point;
executing the test application at the fork point of the test application in a second virtual machine generated using the snapshot at the fork point, wherein the fork point of the test application defines a point in the test application after at least one predetermined input is applied to the test application;
applying a test input to the test application executed in the second virtual machine;
responsive to applying the test input, recording a test output comprising a first output type and a second output type; and
determining that the test input is associated with a bug based at least in part on a distance of an output vector that indicates that the test output is associated with a bug, the distance of the output vector being determined using a first output differential between the first output type of the test output and a corresponding first output type of a happy path output associated with a happy path input at the fork point and a second output differential between the second output type of the test output and a corresponding second output type of the happy path output, wherein the happy path input is an input value that provides a desired or error-free execution of the test application.
2. The method of claim 1 , further comprising:
applying the happy path input to the test application executed in the first virtual machine at the fork point; and
responsive to applying the happy path input, recording the happy path output.
3. The method of claim 1 , further comprising:
executing the test application in a third virtual machine generated using the snapshot at the fork point;
applying a second test input to the test application executed in the third virtual machine; and
responsive to applying the second test input, recording a second test output, wherein the test output is a first test output.
4. The method of claim 3 , further comprising:
analyzing the first test output and the second test output to determine that both the first test output and the second test output belong in an output group.
5. The method of claim 4 , wherein:
analyzing the first test output and the second test output comprises performing k-means clustering analysis and determining that both the first test output and the second test output belong in the output group.
6. The method of claim 1 , wherein:
both the test output and the happy path output include graphical data; and
determining the first output differential comprises obtaining a first Hamming distance between a first perceptual hash associated with the test output and a second Hamming distance between a second perceptual hash associated with the happy path output.
7. The method of claim 1 , wherein:
both the test output and the happy path output include log file data; and
determining the first output differential comprises obtaining a log file differential size difference between a first log file associated with the test output and a second log file associated with the happy path output.
8. The method of claim 1 , wherein:
both the test output and the happy path output include stack trace data; and
determining the first output differential comprises determining a stack trace differential.
9. The method of claim 1 , wherein:
both the test output and the happy path output include keyword count data; and
determining the first output differential comprises determining a keyword count.
10. A system for detecting bugs in an application, the system comprising:
at least one computing device; and
a memory storing instructions that, when executed by the at least one computing device, causes the at least one computing device to:
execute a test application in a first virtual machine;
create a snapshot associated with the first virtual machine and the test application at a fork point;
execute the test application at the fork point of the test application in a second virtual machine generated using the snapshot at the fork point, wherein the fork point of the test application defines a point in the test application after at least one predetermined input is applied to the test application;
apply a test input to the test application executed in the second virtual machine;
responsive to applying the test input, record a test output comprising a first output type and a second output type; and
determine that the test input is associated with a bug based at least in part on a distance of an output vector that indicates that the test output is associated with a bug, the distance of the output vector being determined using a first output differential between the first output type of the test output and a corresponding first output type of a happy path output associated with a happy path input at the fork point and a second output differential between the second output type of the test output and a corresponding second output type of the happy path output, wherein the happy path input is an input value that provides a desired or error-free execution of the test application.
11. The system of claim 10 , wherein when executed the instructions further cause the at least one computing device to:
apply the happy path input to the test application executed in the first virtual machine at the fork point; and
responsive to applying the happy path input, record the happy path output.
12. The system of claim 10 , wherein when executed the instructions further cause the at least one computing device to:
executing the test application in a third virtual machine generated using the snapshot at the fork point;
apply a second test input to the test application executed in the third virtual machine, wherein the test input is a first test input; and
responsive to applying the second test input, record a second test output, wherein the test output is a first test output.
13. The system of claim 12 , wherein when executed the instructions further cause the at least one computing device to analyze the first test output and the second test output to determine that both the first test output and the second test output belong in an output group.
14. The system of claim 13 , wherein when executed the instructions further cause the at least one computing device to analyze the first test output and the second test output comprises performing k-means clustering analysis and determining that both the first test output and the second test output belong in the output group.
15. The system of claim 10 , wherein:
both the test output and the happy path output include graphical data; and
the instructions further cause the at least one computing device to determine the first output differential by obtaining a first Hamming distance between a first perceptual hash associated with the test output and a second Hamming distance between a second perceptual hash associated with happy path output.
16. The system of claim 10 , wherein:
both the test output and the happy path output include log file data; and
the instructions further cause the at least one computing device to determine the first output differential by obtaining a log file differential size difference between a first log file associated with the test output and a second log file associated with the happy path output.
17. The system of claim 10 , wherein:
both the test output and the happy path output include stack trace data; and
the instructions further cause the at least one computing device to determine the first output differential using a stack trace differential.
18. The system of claim 10 , wherein:
both the test output and the happy path output include keyword count data; and
the instructions further cause the at least one computing device to determine the first output differential using a keyword count.
19. The system of claim 10 , wherein the first virtual machine and the second virtual machine are executed in a same computing device of the at least one computing device.
20. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to:
execute a test application in a first virtual machine;
create a snapshot associated with the first virtual machine and the test application at a fork point;
execute the test application at the fork point of the test application in a second virtual machine generated using the snapshot at the fork point, wherein the fork point of the test application defines a point in the test application after at least one predetermined input is applied to the test application;
apply a test input to the test application executed in the second virtual machine;
responsive to applying the test input, record a test output comprising a first output type and a second output type; and
determine that the test input is associated with a bug based at least in part on a distance of an output vector that indicates that the test output is associated with a bug, the distance of the output vector being determined using a first output differential between the first output type of the test output and a corresponding first output type of a happy path output associated with a happy path input at the fork point and a second output differential between the second output type of the test output and a corresponding second output type of the happy path output, wherein the happy path input is an input value that provides a desired or error-free execution of the test application.