Software and framework for reusable automated testing of computer software systems

1. A method of reusing automated test procedures in a computer processing system, the method comprising:

obtaining a plurality of available automated test procedures by calling a static method on a TestCase object to obtain a plurality of methods for testing, the available automated test procedures comprising existing tests that may be applicable to new features or new configurations;

obtaining a plurality of test options to be tested, each option of the plurality of test options comprising option data including an option value; and

for each combination of a current option of the plurality of test options and a current procedure of the plurality of available automated test procedures:

obtaining, by a test suite generator, an annotation for the current procedure, the annotation defining test options or other conditions for which the current procedure is applicable, wherein the annotation does not affect operation of the current procedure;

evaluating, by the test suite generator, the annotation with respect to the option data corresponding to the current option to determine whether the current procedure is applicable for the current option; and

if the current procedure is applicable for the current option:

generating a test specification corresponding to the combination of the current option and the current procedure, the test specification comprising the option data corresponding to the current option and procedure data corresponding to the current procedure;

adding the test specification corresponding to the combination of the current option and the current procedure to a suite of test specifications to be executed, wherein execution of the test specification results in the procedure data corresponding to the current procedure being executed with respect to the option value of the current option, wherein

generating the test specification comprises:

 creating an instance of the TestCase object corresponding to the current procedure; and

 mapping the instance of the TestCase object to the option data and the current procedure data; and

execution of the test specification comprises:

 setting a current test option of the instance of the TestCase object to the option value; and

 executing the current procedure implemented by the TestCase object.

2. The method of claim 1 , wherein obtaining the plurality of available test procedures comprises searching for procedure implementations in a software object.

3. The method of claim 1 , wherein each of the test options comprises a type of software object to test.

4. The method of claim 3 , wherein generating the plurality of test specifications comprises determining which of the available automated test procedures are applicable to each of the types of software objects to test.

5. The method of claim 1 , wherein the plurality of test options comprise the new features or the new configurations.

6. The method of claim 1 , wherein the plurality of test options comprise extensions, bug fixes, and modifications.

7. The method of claim 1 , wherein the plurality of test options correspond to types of reports generated by a multi-tenant database system.

8. The method of claim 1 , wherein the plurality of test options correspond to per-tenant configurable options.

9. The method of claim 1 , further comprising:

selecting the test specification from the suite;

obtaining the option value corresponding to the option associated with the selected test specification; and

executing the procedure data associated with the selected test specification, wherein the procedure data uses the option value to return a test result.

10. The method of claim 1 , wherein:

the plurality of available automated test procedures comprise a plurality of methods for testing charts; and

the plurality of test options comprise different chart types.

11. The method of claim 1 , wherein:

the option value comprises a parameter to direct the execution of the procedure data; and

the execution of the procedure data comprises passing the option value as the parameter to the current procedure at runtime.

12. The method of claim 1 , wherein:

generating the test specification comprises:

creating an instance of a test object corresponding to the current procedure; and

mapping the instance of the test object to the option data and the procedure data; and

execution of the test specification comprises:

setting a current test option of the instance of the test object to the option value; and

executing the current procedure implemented by the test object.

13. The method of claim 1 , wherein each option of the plurality of test options comprises a subclass of a parent class in an object-oriented programming system.

14. The method of claim 1 , wherein each option of the plurality of test options comprises a different report type or display option.

15. A multi-tenant on-demand database system comprising processing hardware and storage, wherein the processing hardware is configured to:

obtain a plurality of available automated test procedures by calling a static method on a TestCase object to obtain a plurality of methods for testing, the available automated test procedures comprising existing tests that may be applicable to new features or new configurations;

obtain a plurality of test options to be tested, each option of the plurality of test options comprising option data including an option value; and

for each combination of a current option of the plurality of test options and a current procedure of the plurality of available automated test procedures:

obtain an annotation for the current procedure, the annotation defining test options or other conditions for which the current procedure is applicable, wherein the annotation does not affect operation of the current procedure;

evaluate the annotation with respect to the option data corresponding to the current option to determine whether the current procedure is applicable for the current option; and

if the current procedure is applicable for the current option:

generate a test specification corresponding to the combination of the current option and the current procedure by creating an instance of the TestCase object corresponding to the current procedure and mapping the instance of the TestCase object to the option data and the current procedure data, the test specification comprising the option data corresponding to the current option and procedure data corresponding to the current procedure;

add the test specification corresponding to the combination of the current option and the current procedure to a suite of test specifications to be executed, wherein execution of the test specification comprises setting a current test option of the instance of the TestCase object to the option value and executing the current procedure implemented by the TestCase object, and execution of the test specification results in the procedure data corresponding to the current procedure being executed with respect to the option value of the current option.

16. A non-transitory media capable of storing program code executable by a computer processing system to:

obtain a plurality of available automated test procedures by calling a static method on a TestCase object to obtain a plurality of methods for testing, the available automated test procedures comprising existing tests that may be applicable to new features or new configurations;

obtain a plurality of test options to be tested, each option of the plurality of test options comprising option data including an option value; and

generate a suite of test specifications to be executed by:

for each combination of a current option of the plurality of test options and a current procedure of the plurality of available automated test procedures:

obtaining an annotation for the current procedure, the annotation defining test options or other conditions for which the current procedure is applicable, wherein the annotation does not affect operation of the current procedure;

evaluating the annotation with respect to the option data corresponding to the current option to determine whether the current procedure is applicable for the current option; and

if the current procedure is applicable for the current option:

generating a test specification corresponding to the combination of the current option and the current procedure, the test specification comprising the option data corresponding to the current option and procedure data corresponding to the current procedure;

adding the test specification corresponding to the combination of the current option and the current procedure to the suite of test specifications, wherein execution of the test specification results in the procedure data corresponding to the current procedure being executed with respect to the option value of the current option, wherein:

 generating the test specification comprises:

 creating an instance of the TestCase object corresponding to the current procedure; and

 mapping the instance of the TestCase object to the option data and the current procedure data; and

 execution of the test specification comprises:

 setting a current test option of the instance of the TestCase object to the option value; and

 executing the current procedure implemented by the TestCase object.

17. The non-transitory media of claim 16 , wherein the program code is executable by the computer processing system to evaluate the annotation with respect to the option to determine whether the procedure is applicable for the option by comparing a name of the procedure to a value of the option.

18. The non-transitory media of claim 16 , wherein the program code is executable by the computer processing system to obtain the plurality of available test procedures by searching for procedure implementations in a software object.

19. The non-transitory media of claim 16 , wherein:

the computer processing system comprises a multi-tenant on-demand database system; and

the plurality of test options correspond to types of reports generated by the multi-tenant on-demand database system.

20. The non-transitory media of claim 16 , wherein:

the computer processing system comprises a multi-tenant on-demand database system; and

the plurality of test options correspond to per-tenant configurable options.