IP Library › Granted Patent US 10,324,831
Granted Patent B1
US 10,324,831 · App. 15/235,385 · Granted Jun 18, 2019

Automated test coverage analysis, execution and reporting

Inventors: Zhipeng Zhang (Shanghai, CN); Shouyuan Cheng (Shanghai, CN); Binbin Deng (Shanghai, CN); Bo Wu (Shanghai, CN); Binhua Lu (Shanghai, CN); Scott D. Von Rhee (Boston, MA)
Assignee: EMC IP Holding Company LLC
G06F11/3692G06F11/3684G06F11/3688
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,324,831
App. No.
15/235,385
Filed
Aug 12, 2016
Granted
Jun 18, 2019
Kind
B1
Art Unit
2192
USPC
717/131
Abstract

Described are techniques for testing software. The techniques may include identifying, at a first point in time, first code that has been modified, identifying, using first mapping information, a testing set of one or more test cases wherein the first mapping information identifies each test case of the testing set as a test case used to test the first code, running the testing set, generating coverage information in accordance with executing; analyzing the coverage information, generating second mapping information in accordance with said analyzing, and updating the first mapping information in accordance with the second mapping information.

Claims (76)

1. A computer-implemented method of testing software comprising:

receiving, at a current point in time, two points in time each denoting a point in time prior to the current point in time and each denoting a different version of software under test, wherein the two points in time denote a time range prior to the current point in time;

identifying first code of the software under test that has been modified within the time range;

identifying, using first mapping information, a testing set of one or more test cases, the first mapping information identifying each test case of the testing set as a test case used to test the first code, wherein said first mapping information is generated based on runtime code coverage information obtained from prior execution of the testing set, wherein a first version of one code entity is used in the software under test at a first of the two points in time and a second version of said one code entity is used in the software under test at a second of the two points in time, wherein said first mapping information indicates that the first version of said one code entity is covered by a first test case and a second test case and indicates that the second version of said one code entity is covered by the first test case but not the second test case;

running the testing set, wherein said running includes executing the first code in accordance with the testing set;

generating coverage information in accordance with said executing the first code;

generating second mapping information in accordance with the coverage information generated as a result of executing the first code;

identifying, in accordance with the second mapping information, a first code entity of the first code that is not covered by at least one test case, the second mapping information indicating that there is no current test case for testing the first code entity;

adding a new test case to test the first code entity;

executing code in accordance with the new test case;

generating second coverage information in accordance with said executing in accordance with the new test case;

generating third mapping information in accordance with the second coverage information; and

updating the first mapping information in accordance with the third mapping information, said third mapping information identifying the new test as causing execution of the first code entity.

2. The method of claim 1 , wherein the first code includes any of a code module, a routine, a function, multiple code modules, multiple functions, and multiple routines.

3. The method of claim 1 , wherein the coverage information identifies what one or more portions of code are executed in said executing the first code.

4. The method of claim 1 , wherein the first code has been modified by performing any of removing a code portion, adding a new code portion, modifying an existing code portion, and adding a new code entity that is any of a module, routine and function.

5. The method of claim 1 , further comprising:

identifying any test case of the testing set that failed during said executing the first code.

6. The method of claim 1 , further comprising:

checking in the first code that includes modified code, wherein responsive to said checking in, first processing is performed, said first processing including:

said identifying the testing set;

said executing the first code;

said generating the coverage information;

analyzing the coverage information;

said generating the second mapping information; and

updating the first mapping information in accordance with the second mapping information.

7. The method of claim 1 , wherein the first mapping information is included in a mapping repository, the first code is included in a code repository, and each of the test cases of the testing set is retrieved from a test case repository.

8. The method of claim 1 , wherein the first code, when executed, performs a data storage operation including any of provisioning storage for a logical device and configuring a RAID (redundant array of independent disks) group, and wherein a third test case of the testing set includes one or more inputs used to perform the data storage operation.

9. A non-transitory computer readable medium comprising code stored thereon that, when executed, performs a method of testing software comprising:

receiving, at a current point in time, two points in time each denoting a point in time prior to the current point in time and each denoting a different version of software under test, wherein the two points in time denote a time range prior to the current point in time;

identifying first code of the software under test that has been modified within the time range;

identifying, using first mapping information, a testing set of one or more test cases, the first mapping information identifying each test case of the testing set as a test case used to test the first code, wherein said first mapping information is generated based on runtime code coverage information obtained from prior execution of the testing set, wherein a first version of one code entity is used in the software under test at a first of the two points in time and a second version of said one code entity is used in the software under test at a second of the two points in time, wherein said first mapping information indicates that the first version of said one code entity is covered by a first test case and a second test case and indicates that the second version of said one code entity is covered by the first test case but not the second test case;

running the testing set, wherein said running includes executing the first code in accordance with the testing set;

generating coverage information in accordance with said executing the first code;

generating second mapping information in accordance with the coverage information generated as a result of executing the first code;

identifying, in accordance with the second mapping information, a first code entity of the first code that is not covered by at least one test case, the second mapping information indicating that there is no current test case for testing the first code entity;

adding a new test case to test the first code entity;

executing code in accordance with the new test case;

generating second coverage information in accordance with said executing in accordance with the new test case;

generating third mapping information in accordance with the second coverage information; and

updating the first mapping information in accordance with the third mapping information, said third mapping information identifying the new test as causing execution of the first code entity.

10. The non-transitory computer readable medium of claim 9 , wherein the first code includes any of a code module, a routine, a function, multiple code modules, multiple functions, and multiple routines.

11. The non-transitory computer readable medium of claim 9 , wherein the coverage information identifies what one or more portions of code are executed in said executing the first code.

12. The non-transitory computer readable medium of claim 9 , wherein the first code has been modified by performing any of removing a code portion, adding a new code portion, modifying an existing code portion, and adding a new code entity that is any of a module, routine and function.

13. The non-transitory computer readable medium of claim 9 , wherein the method further comprises:

identifying any test case of the testing set that failed during said executing the first code.

14. The non-transitory computer readable medium of claim 9 , wherein the method further comprises:

checking in the first code that includes modified code, wherein responsive to said checking in, first processing is performed, said first processing including:

said identifying the testing set;

said executing the first code;

said generating the coverage information;

analyzing the coverage information;

said generating the second mapping information; and

updating the first mapping information in accordance with the second mapping information.

15. The non-transitory computer readable medium of claim 9 , wherein the first mapping information is included in a mapping repository, the first code is included in a code repository, and each of the test cases of the testing set is retrieved from a test case repository.

16. The non-transitory computer readable medium of claim 9 , wherein the first code, when executed, performs a data storage operation including any of provisioning storage for a logical device and configuring a RAID (redundant array of independent disks) group, and wherein a first third test case of the testing set includes one or more inputs used to perform the data storage operation.

17. A system comprising:

a processor;

a memory comprising code stored therein that, when executed by a processor, performs a method comprising:

receiving, at a current point in time, two points in time each denoting a point in time prior to the current point in time and each denoting a different version of software under test, wherein the two points in time denote a time range prior to the current point in time;

identifying first code of the software under test that has been modified within the time range;

identifying, using first mapping information, a testing set of one or more test cases, the first mapping information identifying each test case of the testing set as a test case used to test the first code, wherein said first mapping information is generated based on runtime code coverage information obtained from prior execution of the testing set, wherein a first version of one code entity is used in the software under test at a first of the two points in time and a second version of said one code entity is used in the software under test at a second of the two points in time, wherein said first mapping information indicates that the first version of said one code entity is covered by a first test case and a second test case and indicates that the second version of said one code entity is covered by the first test case but not the second test case;

running the testing set, wherein said running includes executing the first code in accordance with the testing set;

generating coverage information in accordance with said executing the first code;

generating second mapping information in accordance with the coverage information generated as a result of executing the first code;

identifying, in accordance with the second mapping information, a first code entity of the first code that is not covered by at least one test case, the second mapping information indicating that there is no current test case for testing the first code entity;

adding a new test case to test the first code entity;

executing code in accordance with the new test case;

generating second coverage information in accordance with said executing in accordance with the new test case;

generating third mapping information in accordance with the second coverage information; and

updating the first mapping information in accordance with the third mapping information, said third mapping information identifying the new test as causing execution of the first code entity.

18. The system of claim 17 , wherein the first code includes any of a code module, a routine, a function, multiple code modules, multiple functions, and multiple routines.

19. The system of claim 17 , wherein the coverage information identifies what one or more portions of code are executed in said executing the first code.

20. The method of claim 1 further comprising:

identifying an existing test case without mapping information; executing the existing test case; and

generating, in accordance with said executing the existing test case, additional mapping information identifying one or more code entities tested by the existing test case.

Assignments (9)
RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (053546/0001) Recorded Jun 23, 2022
From: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
To: DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.); DELL INTERNATIONAL L.L.C.; DELL PRODUCTS L.P.; DELL USA L.P.; EMC CORPORATION; DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.); EMC IP HOLDING COMPANY LLC
Reel/Frame 071642/0001 →
RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001) Recorded May 20, 2022
From: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
To: DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO ASAP SOFTWARE EXPRESS, INC.); DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.); DELL USA L.P.; DELL INTERNATIONAL L.L.C.; DELL PRODUCTS L.P.; DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.); EMC CORPORATION (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MAGINATICS LLC); EMC IP HOLDING COMPANY LLC (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MOZY, INC.); SCALEIO LLC
Reel/Frame 061753/0001 →
RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001) Recorded Apr 26, 2022
From: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
To: DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO ASAP SOFTWARE EXPRESS, INC.); DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.); DELL USA L.P.; DELL INTERNATIONAL L.L.C.; DELL PRODUCTS L.P.; DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.); EMC CORPORATION (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MAGINATICS LLC); EMC IP HOLDING COMPANY LLC (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MOZY, INC.); SCALEIO LLC
Reel/Frame 061324/0001 →
SECURITY AGREEMENT Recorded Apr 22, 2020
From: CREDANT TECHNOLOGIES INC.; DELL INTERNATIONAL L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL USA L.P.; EMC CORPORATION; FORCE10 NETWORKS, INC.; WYSE TECHNOLOGY L.L.C.; EMC IP HOLDING COMPANY LLC
To: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A.
Reel/Frame 053546/0001 →
SECURITY AGREEMENT Recorded Mar 21, 2019
From: CREDANT TECHNOLOGIES, INC.; DELL INTERNATIONAL L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL USA L.P.; EMC CORPORATION; FORCE10 NETWORKS, INC.; WYSE TECHNOLOGY L.L.C.; EMC IP HOLDING COMPANY LLC
To: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A.
Reel/Frame 049452/0223 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 29, 2016
From: EMC CORPORATION
To: EMC IP HOLDING COMPANY LLC
Reel/Frame 040203/0001 →
SECURITY AGREEMENT Recorded Sep 21, 2016
From: ASAP SOFTWARE EXPRESS, INC.; AVENTAIL LLC; CREDANT TECHNOLOGIES, INC.; DELL USA L.P.; DELL INTERNATIONAL L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL SOFTWARE INC.; DELL SYSTEMS CORPORATION; EMC CORPORATION; EMC IP HOLDING COMPANY LLC; FORCE10 NETWORKS, INC.; MAGINATICS LLC; MOZY, INC.; SCALEIO LLC; SPANNING CLOUD APPS LLC; WYSE TECHNOLOGY L.L.C.
To: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
Reel/Frame 040136/0001 →
SECURITY AGREEMENT Recorded Sep 21, 2016
From: ASAP SOFTWARE EXPRESS, INC.; AVENTAIL LLC; CREDANT TECHNOLOGIES, INC.; DELL USA L.P.; DELL INTERNATIONAL L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL SOFTWARE INC.; DELL SYSTEMS CORPORATION; EMC CORPORATION; EMC IP HOLDING COMPANY LLC; FORCE10 NETWORKS, INC.; MAGINATICS LLC; MOZY, INC.; SCALEIO LLC; SPANNING CLOUD APPS LLC; WYSE TECHNOLOGY L.L.C.
To: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH, AS COLLATERAL AGENT
Reel/Frame 040134/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 12, 2016
From: ZHANG, ZHIPENG; CHENG, SHOUYUAN; DENG, BINBIN; WU, BO; LU, BINHUA; VON RHEE, SCOTT D.
To: EMC CORPORATION
Reel/Frame 039415/0315 →
Continuity (1)
Continuation 14314411 · Jun 25, 2014
Cited By (2)
US 12,511,207 US 12,664,006