IP Library › Granted Patent US 12,072,790
Granted Patent B1
US 12,072,790 · App. 17/224,904 · Granted Aug 27, 2024

Mutation testing within continuous integration systems

Inventors: Andrew L Pearson (Normal, IL); Nate Shepherd (Bloomington, IL)
Assignee: State Farm Mutual Automobile Insurance Company
G06F11/3688G06F11/3676G06F11/3684G06F11/3692
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,072,790
App. No.
17/224,904
Granted
Aug 27, 2024
Kind
B1
Abstract

Techniques described herein relate to implementing mutation testing of software applications associated with continuous integration (CI) systems. A mutation test system may determine one or more portions of modified source code within an application codebase. Mutated applications may be generated based on the modified source code, and a mutation test system may determine subsets application test suites for execution based on the portions of the modified source code and/or other factors. In various examples, the mutation test system may use mappings between portions of source code and test subsets, and/or machine-learned models or heuristics-based techniques to determine subsets of test suites based on discreet source code modifications. Mutation testing can be performed by executing the determined test subsets on the mutated applications, and the results may be used by the CI system to control the integration of the code changes into the shared source code repository and/or automated testing of the application build.

Claims (90)

1. A computer-implemented method, comprising:

receiving, by a computer system, a request to integrate a source code change into a codebase associated with an application; determining an application test suite associated with the application;

determining a first line of source code associated with the requested source code change;

analyzing, by the computer system, the first line of source code to determine a first source code attribute associated with the first line of source code, wherein the first source code attribute includes at least one of (a) a type of programming statement in the first line of source code, (b) a type of operator in the first line of source code, (c) an indication of a type of function invoked by the first line of source code, (d) an indication of whether the first line of source code is within a conditional code block, or (e) an indication of whether the first line of source code is within a programming loop;

determining a first test subset of the application test suite, based at least in part on the first source code attribute associated with the first line of source code, wherein determining the first test subset comprises inputting the first source code attribute into a trained model configured to output data identifying a subset of the test suite;

mutating, by the computer system, the source code change into a mutated source code change;

building, by the computer system, a mutated application by compiling one or more software classes including the mutated source code change; and

executing, by the computer system, the first test subset of the application test suite on the mutated application.

2. The computer-implemented method of claim 1 , further comprising:

determining that the source code change modifies a second line of source code within the codebase, wherein the second line of source code is within a different function from the first line of source code;

determining a second set of tests associated with the second line of source code, wherein the second set of tests is a second subset of the test suite different from the first subset; and

executing the second set of tests on the mutated application.

3. The computer-implemented method of claim 1 , further comprising:

initiating an operation to integrate the source code change into the codebase, based at least in part on test results associated with executing the first test subset.

4. The computer-implemented method of claim 1 , wherein determining the first test subset for execution on the mutated application comprises:

retrieving test results associated with a previous execution of the test suite, on a previous mutated application associated with the first line of source code; and

determining, within the test results associated with a previous execution of the test suite, one or more passing tests; and

excluding the one or more passing tests from the first subset of the test suite associated with the first line of source code.

5. The computer-implemented method of claim 1 , wherein determining the first test subset for execution on the mutated application comprises:

accessing a mapping storing associations between different subsets of the test suite, and corresponding lines of source code within the codebase.

6. The computer-implemented method of claim 1 , wherein:

mutating the source code change is based on a set of code mutation rules; and

determining the first test subset for execution on the mutated application, is based at least in part on the set of code mutation rules.

7. The computer-implemented method of claim 1 , further comprising:

receiving a mutation test confidence level associated with the request to integrate the source code change; and

determining the first test subset for execution on the mutated application, based at least in part on the mutation test confidence level.

8. One or more non-transitory computer-readable media storing instructions executable by a processor, wherein the instructions, when executed by the processor, cause the processor to perform operations comprising:

receiving a request to integrate a source code change into a codebase associated with an application;

determining an application test suite associated with the application;

determining a first line of source code associated with the requested source code change;

analyzing the first line of source code to determine a first source code attribute associated with the first line of source code, wherein the first source code attribute includes at least one of (a) a type of programming statement in the first line of source code, (b) a type of operator in the first line of source code, (c) an indication of a type of function invoked by the first line of source code, (d) an indication of whether the first line of source code is within a conditional code block, or (e) an indication of whether the first line of source code is within a programming loop;

determining a first test subset of the application test suite, based at least in part on the first source code attribute associated with the first line of source code, wherein determining the first test subset comprises inputting the first source code attribute into a trained model configured to output data identifying a subset of the test suite;

mutating the source code change into a mutated source code change;

building a mutated application based on the mutated source code change; and

executing the first test subset of the application test suite on the mutated application.

9. The one or more non-transitory computer-readable media of claim 8 , the operations further comprising:

determining that the source code change affects a second line of source code within the codebase, wherein the second line of source code is within a different function from the first line of source code;

determining a second set of tests associated with the second line of source code, wherein the second set of tests is a second subset of the test suite different from the first subset; and

executing the second set of tests on the mutated application.

10. The one or more non-transitory computer-readable media of claim 8 , wherein determining the first test subset comprises:

accessing a mapping storing associations between different subsets of the test suite, and at least one of corresponding lines of source code or corresponding functions within the codebase.

11. The one or more non-transitory computer-readable media of claim 8 , wherein:

mutating the source code change is based on a set of code mutation rules; and

determining the first test subset is based at least in part on the set of code mutation rules.

12. A computer-implemented method, comprising:

receiving, by a computer system, a request to integrate a source code change into a codebase associated with an application; determining an application test suite associated with the application;

determining a first line of source code associated with the requested source code change;

analyzing, by the computer system, the first line of source code to determine a first source code attribute associated with the first line of source code, wherein the first source code attribute includes at least one of (a) a type of programming statement in the first line of source code, (b) a type of operator in the first line of source code, (c) an indication of a type of function invoked by the first line of source code, (d) an indication of whether the first line of source code is within a conditional code block, or (e) an indication of whether the first line of source code is within a programming loop;

determining a first test subset of the application test suite, based at least in part on the first source code attribute associated with the first line of source code;

mutating, by the computer system, the source code change into a mutated source code change, wherein mutating the source code change comprises:

determining one or more code mutation rules, based at least in part on the first source code attribute associated with the first line of source code; and

modifying the source code change, using the code mutation rules, into the mutated source code change;

building, by the computer system, a mutated application by compiling one or more software classes including the mutated source code change; and

executing, by the computer system, the first test subset of the application test suite on the mutated application.

13. The computer-implemented method of claim 12 , further comprising:

determining that the source code change modifies a second line of source code within the codebase, wherein the second line of source code is within a different function from the first line of source code;

determining a second set of tests associated with the second line of source code, wherein the second set of tests is a second subset of the test suite different from the first subset; and

executing the second set of tests on the mutated application.

14. The computer-implemented method of claim 12 , wherein determining the first test subset for execution on the mutated application comprises:

retrieving test results associated with a previous execution of the test suite, on a previous mutated application associated with the first line of source code; and

determining, within the test results associated with a previous execution of the test suite, one or more passing tests; and

excluding the one or more passing tests from the first subset of the test suite associated with the first line of source code.

15. The computer-implemented method of claim 12 , wherein determining the first test subset for execution on the mutated application comprises:

accessing a mapping storing associations between different subsets of the test suite, and corresponding lines of source code within the codebase.

16. The computer-implemented method of claim 12 , wherein:

mutating the source code change is based on a set of code mutation rules; and

determining the first test subset for execution on the mutated application, is based at least in part on the set of code mutation rules.

17. The computer-implemented method of claim 12 , further comprising:

receiving a mutation test confidence level associated with the request to integrate the source code change; and

determining the first test subset for execution on the mutated application, based at least in part on the mutation test confidence level.

18. A computer system, comprising:

one or more processors; and

one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

receiving, by the computer system, a request to integrate a source code change into a codebase associated with an application; determining an application test suite associated with the application;

determining a first line of source code associated with the requested source code change;

analyzing, by the computer system, the first line of source code to determine a first source code attribute associated with the first line of source code, wherein the first source code attribute includes at least one of (a) a type of programming statement in the first line of source code, (b) a type of operator in the first line of source code, (c) an indication of a type of function invoked by the first line of source code, (d) an indication of whether the first line of source code is within a conditional code block, or (e) an indication of whether the first line of source code is within a programming loop;

determining a first test subset of the application test suite, based at least in part on the first source code attribute associated with the first line of source code;

mutating, by the computer system, the source code change into a mutated source code change, wherein mutating the source code change comprises:

determining one or more code mutation rules, based at least in part on the first source code attribute associated with the first line of source code; and

modifying the source code change, using the code mutation rules, into the mutated source code change;

building, by the computer system, a mutated application by compiling one or more software classes including the mutated source code change; and

executing, by the computer system, the first test subset of the application test suite on the mutated application.

19. The computer system of claim 18 , the operations further comprising:

determining that the source code change modifies a second line of source code within the codebase, wherein the second line of source code is within a different function from the first line of source code;

determining a second set of tests associated with the second line of source code, wherein the second set of tests is a second subset of the test suite different from the first subset; and

executing the second set of tests on the mutated application.

20. The computer system of claim 18 , wherein determining the first test subset for execution on the mutated application comprises:

retrieving test results associated with a previous execution of the test suite, on a previous mutated application associated with the first line of source code; and

determining, within the test results associated with a previous execution of the test suite, one or more passing tests; and

excluding the one or more passing tests from the first subset of the test suite associated with the first line of source code.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 8, 2021
From: PEARSON, ANDREW L; SHEPHERD, NATE
To: STATE FARM MUTUAL AUTOMOBILE INSURANCE COMPANY
Reel/Frame 055868/0618 →
Cited By (7)
US 12,222,847 US 12,248,382 US 12,423,218 US 12,481,944 US 12,517,807 US 12,585,543 US 12,711,238