Vendor assisted customer individualized testing
In an example embodiment, first a system is provided that enables tests created by a second entity to run with first entity data and configurations. A test module is able to assess test results with dynamic input (e.g., rule-based assessments). Second, a system is provided to run and test these test modules on the first entity system with first entity configuration and let the first entity define the test input and interpret the test results. This is not performed on all available tests, but only on the tests that are related to the change to be implemented.
1. A system comprising:
at least one hardware processor; and
a non-transitory computer-readable medium storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising:
obtaining a first data structure containing a map directly relating software artifacts to test modules;
obtaining one or more test modules, each test module containing code that, when executed, performs a test on an installed first version of a software program on a computing device;
identifying a first software artifact that is in a call stack history of the computing device for the first version of the software program and that is contained in the map;
searching for any other software artifacts that were more recently used during execution of a call in the call stack history than the first software artifact, and that are also contained in the map; and
executing a test module specified in the map for each artifact in a first set of one or more software artifacts including the first software artifact and the any other identified software artifacts, producing a first set of test results.
2. The system of claim 1 , wherein each test module includes a set of test rules, the test rules executed to determine whether results of a corresponding test are positive or negative.
3. The system of claim 2 , wherein the operations further comprise receiving an adjustment to the set of test rules, from a user, via a graphical user interface.
4. The system of claim 3 , wherein the operations further comprise writing the adjustment of the set of test rules to a database and automatically rerunning a corresponding test module using the adjusted set of test rules.
5. The system of claim 1 , wherein the first set of one or more software artifacts does not include any software artifacts that are contained in modules that have not been previously used by a first entity.
6. The system of claim 1 , wherein the executing a test module further includes executing a test module specified in the map for each artifact that is contained in a where-used list on the computing device, as having used an artifact that is contained in a list of software artifacts that have changed but that is not contained in the map.
7. The system of claim 1 , wherein the executing includes executing the test module on user data.
8. A computer-implemented method comprising:
obtaining a first data structure containing a map directly relating software artifacts to test modules;
obtaining one or more test modules, each test module containing code that, when executed, performs a test on an installed first version of a software program on a computing device;
identifying a first software artifact that is in a call stack history of the computing device for the first version of the software program and that is contained in the map;
searching for any other software artifacts that were more recently used during execution of a call in the call stack history than the first software artifact, and that are also contained in the map; and
executing a test module specified in the map for each artifact in a first set of one or more software artifacts including the first software artifact and the any other identified software artifacts, producing a first set of test results.
9. The computer-implemented method of claim 8 , wherein each test module includes a set of test rules, the test rules executed to determine whether results of a corresponding test are positive or negative.
10. The computer-implemented method of claim 9 , further comprising receiving an adjustment to the set of test rules, from a user, via a graphical user interface.
11. The computer-implemented method of claim 10 , further comprising writing the adjustment of the set of test rules to a database and automatically rerunning a corresponding test module using the adjusted set of test rules.
12. The computer-implemented method of claim 8 , wherein the first set of one or more software artifacts does not include any software artifacts that are contained in modules that have not been previously used by a first entity.
13. The computer-implemented method of claim 8 , wherein the executing a test module further includes executing a test module specified in the map for each artifact that is contained in a where-used list on the computing device, as having used an artifact that is contained in a list of software artifacts that have changed but that is not contained in the map.
14. The computer-implemented method of claim 8 , wherein the executing includes executing the test module on user data.
15. A non-transitory machine-readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:
obtaining a first data structure containing a map directly relating software artifacts to test modules;
obtaining one or more test modules, each test module containing code that, when executed, performs a test on an installed first version of a software program on a computing device;
identifying a first software artifact that is in a call stack history of the computing device for the first version of the software program and that is contained in the map;
searching for any other software artifacts that were more recently used during execution of a call in the call stack history than the first software artifact, and that are also contained in the map; and
executing a test module specified in the map for each artifact in a first set of one or more software artifacts including the first software artifact and the any other identified software artifacts, producing a first set of test results.
16. The non-transitory machine-readable medium of claim 15 , wherein each test module includes a set of test rules, the test rules executed to determine whether results of a corresponding test are positive or negative.
17. The non-transitory machine-readable medium of claim 16 , wherein the operations further comprise receiving an adjustment to the set of test rules, from a user, via a graphical user interface.
18. The non-transitory machine-readable medium of claim 17 , wherein the operations further comprise writing the adjustment of the set of test rules to a database and automatically rerunning a corresponding test module using the adjusted set of test rules.
19. The non-transitory machine-readable medium of claim 15 , wherein the first set of one or more software artifacts does not include any software artifacts that are contained in modules that have not been previously used by a first entity.
20. The non-transitory machine-readable medium of claim 15 , wherein the executing a test module further includes executing a test module specified in the map for each artifact that is contained in a where-used list on the computing device, as having used an artifact that is contained in a list of software artifacts that have changed but that is not contained in the map.