Automated bisecting tool for regression detection
Arrangements for intelligent regression detection are provided. Initial parameters including a first commit and a second commit may be received and a midpoint may be identified. A test may be executed at the midpoint. A regression detection algorithm may be used to determine whether a regression is present at the midpoint. The regression detection algorithm may automatically identify patterns from test results indicative of regressions and detect regressions based on the identified patterns. The midpoint may be set as the first commit or the second commit based on whether the regression is present at the midpoint. A new midpoint may be identified between the updated initial parameters. A test may be executed to determine whether the regression is present at the new midpoint. A search space of commit points may be automatically iteratively refined using the new midpoint at each iteration until a single commit causing the regression is identified.
1 . A system, comprising:
at least one processor; and
at least one memory storing instructions, which when executed by the at least one processor, result in operations for regression bisecting comprising:
receiving initial parameters including a first commit and a second commit, wherein the first commit and the second commit comprise endpoints of an interval;
identifying a midpoint between the first commit and the second commit, wherein the midpoint represents a commit to be tested;
executing a test of the commit at the midpoint;
determining, using a regression detection algorithm, whether a regression is present at the midpoint, wherein the regression detection algorithm automatically identifies patterns from test results indicative of regressions and detects regressions based on the identified patterns, the identified patterns being associated with a specified symptom occurring at the midpoint when the regression is a functional regression and associated with benchmark samples collected at the midpoint when the regression is a performance regression;
setting the midpoint as the first commit or the second commit based on whether the regression is present at the midpoint, wherein the setting updates the initial parameters;
identifying a new midpoint between the updated initial parameters, wherein the new midpoint represents a new commit to be tested; and
automatically iteratively refining a search space of commit points using the new midpoint at each iteration until a single commit is identified, wherein the single commit corresponds to a commit causing the regression.
2 . The system of claim 1 , wherein the first commit comprises a commit point in a commit history known to not contain the regression.
3 . The system of claim 1 , wherein the second commit comprises a commit point in a commit history known to contain the regression.
4 . The system of claim 1 , further comprising: verifying a state of the first commit and a state of the second commit.
5 . The system of claim 1 , further comprising:
pausing a regression bisecting operation in response to an error; and
continuing the regression bisecting operation from a paused point.
6 . The system of claim 1 , further comprising:
storing regression bisecting decisions in a log; and
replaying the log in subsequent test runs.
7 . The system of claim 1 , further comprising:
automatically calculating a baseline for the first commit.
8 . A computer-implemented method comprising:
receiving initial parameters including a first commit and a second commit, wherein the first commit and the second commit comprise endpoints of an interval;
identifying a midpoint between the first commit and the second commit, wherein the midpoint represents a commit to be tested;
executing a test of the commit at the midpoint;
determining, using a regression detection algorithm, whether a regression is present at the midpoint, wherein the regression detection algorithm automatically identifies patterns from test results indicative of regressions and detects regressions based on the identified patterns, the identified patterns being associated with a specified symptom occurring at the midpoint when the regression is a functional regression and associated with benchmark samples collected at the midpoint when the regression is a performance regression;
setting the midpoint as the first commit or the second commit based on whether the regression is present at the midpoint, wherein the setting updates the initial parameters;
identifying a new midpoint between the updated initial parameters, wherein the new midpoint represents a new commit to be tested; and
automatically iteratively refining a search space of commit points using the new midpoint at each iteration until a single commit is identified, wherein the single commit corresponds to a commit causing the regression.
9 . The computer-implemented method of claim 8 , wherein the first commit comprises a commit point in a commit history known to not contain the regression.
10 . The computer-implemented method of claim 8 , wherein the second commit comprises a commit point in a commit history known to contain the regression.
11 . The computer-implemented method of claim 8 , further comprising:
verifying a state of the first commit and a state of the second commit.
12 . The computer-implemented method of claim 8 , further comprising:
pausing a regression bisecting operation in response to an error; and
continuing the regression bisecting operation from a paused point.
13 . The computer-implemented method of claim 8 , further comprising:
storing regression bisecting decisions in a log; and
replaying the log in subsequent test runs.
14 . The computer-implemented method of claim 8 , further comprising:
automatically calculating a baseline for the first commit.
15 . A non-transitory computer readable medium storing instructions, which when executed by at least one processor, result in operations comprising:
receiving initial parameters including a first commit and a second commit, wherein the first commit and the second commit comprise endpoints of an interval;
identifying a midpoint between the first commit and the second commit, wherein the midpoint represents a commit to be tested;
executing a test of the commit at the midpoint;
determining, using a regression detection algorithm, whether a regression is present at the midpoint, wherein the regression detection algorithm automatically identifies patterns from test results indicative of regressions and detects regressions based on the identified patterns, the identified patterns being associated with a specified symptom occurring at the midpoint when the regression is a functional regression and associated with benchmark samples collected at the midpoint when the regression is a performance regression;
setting the midpoint as the first commit or the second commit based on whether the regression is present at the midpoint, wherein the setting updates the initial parameters;
identifying a new midpoint between the updated initial parameters, wherein the new midpoint represents a new commit to be tested; and
automatically iteratively refining a search space of commit points using the new midpoint at each iteration until a single commit is identified, wherein the single commit corresponds to a commit causing the regression.
16 . The non-transitory computer readable medium of claim 15 , wherein the first commit comprises a commit point in a commit history known to not contain the regression.
17 . The non-transitory computer readable medium of claim 15 , wherein the second commit comprises a commit point in a commit history known to contain the regression.
18 . The non-transitory computer readable medium of claim 15 , further comprising: verifying a state of the first commit and a state of the second commit.