IP Library Granted Patent US 12,229,549
Granted Patent B2
US 12,229,549 · App. 17/964,832 · Granted Feb 18, 2025

Scalable code testing and benchmarking

Inventors: Julian Thome (Esch/Alzette, DE); James Johnson (Sammamish, WA); Isaac Dawson (Yokohama, JP); Dinesh Bolkensteyn (San Francisco, CA); Michael Henriksen (San Francisco, CA); Mark Art (Mosman, AU)
Assignee: GitLab Inc.
G06F8/71G06F8/658G06F11/3604G06F11/3672
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,229,549
App. No.
17/964,832
Granted
Feb 18, 2025
Kind
B2
Abstract

A method or system for testing and benchmarking commits made on source code. The system extracts commits from a history of a first code history that records a sequence of commits made thereon. The extracted commits are then combined into a sequence of patches, each of which includes changes made during consecutive commits. The system also establishes a connection with a system under test (SUT) having a second code repository corresponding to a historical version of the first code repository before the extracted commits were made, and sequentially applies the sequence of the patches to the second code repository. Performance of the SUT is monitored during the application of the sequence of the patches to determine whether the SUT performs as expected after the application of each patch.

Claims (60)

1. A method of testing and benchmarking commits made on source code, the method comprising:

extracting commits from a history of a first code repository, the history recording a sequence of commits made on the first code repository, each commit of the sequence of commits including at least one change made on at least one code file in the first code repository,

wherein extracting commits comprises extracting a subset of commits from the history based on a specified criterion, and

wherein the specified criterion includes an allow list that specifies one or more files that are allowed to be modified, and extracting the subset of commits includes extracting commits that modify at least one file specified in the allow list;

combining the extracted commits into a sequence of patches, each patch including changes made on one or more code files that occurred during consecutive commits, wherein each patch includes a plurality of commits;

establishing a connection with a system under test (SUT) having a second code repository, the second code repository corresponding to a historical version of the first code repository before the extracted commits were made; and

sequentially applying the sequence of patches to the second code repository, sequentially applying the sequence of patches including:

applying a patch in the sequence on the second code repository;

monitoring performance of the SUT after applying the patch on the SUT; and

responsive to determining that the SUT performs as expected after applying the patch, applying a next patch in the sequence to the second code repository.

2. The method of claim 1 , wherein monitoring the performance of the SUT after applying the patch in the sequence comprises:

executing a script configured to collect data associated with the performance of the SUT after the patch is applied; and

determining whether the SUT performs as expected based on the collected data.

3. The method of claim 2 , wherein the data is collected via an API that triggers execution of the script responsive to applying the patch.

4. The method of claim 1 , wherein:

for each commit, the history records a time when the commit is made,

the specified criterion includes a time frame having a starting time and an end time, and

extracting the subset of commits includes extracting commits made within the time frame.

5. The method of claim 1 , wherein extracting commits that modify at least one file specified in the allow list comprises:

for each commit in the history,

generating a first hash based on the one or more files in the allow list before the commit is made;

generating a second hash based on the one or more files after the commit is made;

determining whether the first hash and the second hash are different; and

responsive to determining that the first hash and the second hash are different, determining that the commit modifies at least one file in the allow list; and

responsive to determining that the commit modifies at least one file in the allow list, extracting the commit.

6. The method of claim 1 , wherein combining the extracted commits into a sequence of patches comprises combining N consecutive commits of the extracted commits into a single patch.

7. The method of claim 1 , the method further comprising:

initializing a repository on the SUT as the second code repository based on a historical version of the first code repository from before at least the extracted commits were made.

8. The method of claim 1 , the method further comprising:

initializing the SUT in a container based on a container image.

9. A computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to:

extract commits from a history of a first code repository, the history recording a sequence of commits made on the first code repository, each commit of the sequence of commits including at least one change made on at least one code file in the first code repository,

wherein extracting commits comprises extracting a subset of commits from the history based on a specified criterion, and

wherein the specified criterion includes an allow list that specifies one or more files that are allowed to be modified, and extracting the subset of commits includes extracting commits that modify at least one file specified in the allow list;

combine the extracted commits into a sequence of patches, each patch including changes made on one or more code files that occurred during consecutive commits, wherein each patch includes a plurality of commits;

establish a connection with a system under test (SUT) having a second code repository, the second code repository corresponding to a historical version of the first code repository before the extracted commits were made; and

sequentially apply the sequence of patches to the second code repository, sequentially applying the sequence of patches including:

applying a patch in the sequence on the second code repository;

monitoring performance of the SUT after applying the patch on the SUT; and

responsive to determining that the SUT performs as expected after applying the patch, applying a next patch in the sequence to the second code repository.

10. The computer program product of claim 9 , wherein monitoring performance of the SUT after application of a patch in the sequence comprises:

executing a script configured to collect data associated with performance of the SUT after the patch is applied; and

determining whether the SUT performs as expected based on the collected data.

11. The computer program product of claim 10 , wherein the data is collected via an API that triggers execution of the script responsive to applying the patch.

12. The computer program product of claim 9 , wherein:

for each commit, the history records a time when the commit is made,

the specified criterion includes a time frame having a starting time and an end time, and

extracting the subset of commits includes extracting commits made within the time frame.

13. The computer program product of claim 9 , wherein extracting commits that modify at least one file specified in the allow list comprises:

for each commit in the history,

generating a first hash based on the one or more files in the allow list before the commit is made;

generating a second hash based on the one or more files after the commit is made;

determining whether the first hash and the second hash are different; and

responsive to determining that the first hash and the second hash are different, determining that the commit modifies at least one file in the allow list; and

responsive to determining that the commit modifies at least one file in the allow list, extracting the commit.

14. The computer program product of claim 9 , wherein combining the extracted commits into a sequence of patches comprises combining every N consecutive commits of the extracted commits into a single patch.

15. The computer program product of claim 9 , the non-transitory computer readable storage medium having additional instructions encoded thereon that, when executed by a processor, cause the processor to:

initialize a repository on the SUT as the second code repository based on a historical version of the first code repository from before at least the extracted commits were made.

16. The computer program product of claim 9 , the non-transitory computer readable storage medium having additional instructions encoded thereon that, when executed by a processor, cause the processor to:

initialize the SUT in a container based on a container image.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 2, 2022
From: THOME, JULIAN; JOHNSON, JAMES; DAWSON, ISAAC; BOLKENSTEYN, DINESH; HENRIKSEN, MICHAEL; ART, MARK
To: GITLAB INC.
Reel/Frame 061637/0368 →
Continuity (1)
Related Publication 20240126542A1 · Apr 18, 2024
References Cited (22)
US 7735078B1 · Vaidya · 2010 [cited by examiner]
US 11232097B1 · Antao · 2022 [cited by examiner]
US 11544050B1 · Ankit · 2023 [cited by examiner]
US 20170168798A1 · Lang et al. · 2017 [cited by applicant]
US 20170199737A1 · Bofferding · 2017 [cited by examiner]
US 20170277517A1 · Petitgrand et al. · 2017 [cited by applicant]
US 20180067740A1 · Bofferding et al. · 2018 [cited by applicant]
US 20200250074A1 · Zhang · 2020 [cited by examiner]
US 20210042216A1 · Magnezi · 2021 [cited by examiner]
US 20220091969A1 · Han · 2022 [cited by examiner]
Gitlab, “sourcewarp,” undated, 6 pages, [Online] [Retrieved on May 2, 2023] Retrieved from the internet <URL: https://gitlab.com/gitlab-org/vulnerability-research/foss/sourcewarp>. [cited by applicant]
Gitlab, “Advanced vulnerability tracking,” undated, 19 pages, GitLab 14.2. [Online] [Retrieved on May 2, 2023] Retrieved from the internet <URL: https://docs.gitlab.com/ee/user/application_security/sast/#advanced-vulner… [cited by applicant]
Kamalbasha, S. et al., “Bayesian A/B Testing for Business Decisions,” Data Science—Analytics and Applications: Proceedings of the 3rd International Data Science Conference-iDSC2020, arXiv:2003.02769v1, Mar. 5, 2020, pp.… [cited by applicant]
KPMG, “Agile Transformation,” KPMG Advisory N.V. 2019, pp. 1-42. [cited by applicant]
Li, P. L. et al., “Evolving Software to be ML-Driven Utilizing Real-World A/B Testing: Experiences, Insights, Challenges,” 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Pra… [cited by applicant]
Long, Z. et al., “WebRR: self-replay enhanced robust record/replay for web application testing,” ESEC/FSE 2020: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the … [cited by applicant]
Thome, J. et al., “Search-Based Security Testing of Web Applications,” Proceedings of the 7th International workshop on search-based software testing, Jun. 2, 2014, pp. 5-14. [cited by applicant]
Thummalapenta, S. et al., “Automating Test Automation,” 34th international conference on software engineering (ICSE). IEEE, Jun. 2, 2012, pp. 881-891. [cited by applicant]
Thummalapenta, S. et al., “Efficiently Scripting Change-Resilient Tests,” Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering, Nov. 11, 2012, pp. 1-4. [cited by applicant]
Yandrapally, R. et al., “Robust Test Automation using Contextual Clues,” Proceedings of the 2014 International Symposium on Software Testing and Analysis, Jul. 21, 2014, pp. 304-314. [cited by applicant]
PCT International Search Report and Written Opinion, PCT Application No. PCT/IB2023/055784, Sep. 18, 2023, 8 pages. [cited by applicant]
Hoang, T. et al., “PatchNet: A Tool for Deep Patch Classification,” 2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings, arXiv:1903.02063v2, Mar. 26, 2019, pp. 83-86. [cited by applicant]