Automated algorithmic verification in an embedded complex distributed storage environment
A model of an algorithm to be tested is created based on the steps that the algorithm uses to implement a particular function. The model is used by an algorithm verification process to control execution of the application under test. Stubs (small pieces of code) are inserted into the application under test which are used to control execution of the algorithm to determine the result of execution of the algorithm at various locations. The remote stubs enable the testing framework to control execution of the application, stop execution of the application, view data structures associated with the application, freeze system behavior, induce multi-layered dependent sequences, analyze logs, change system flow based on test requests, examine internal data structures as part of the tests, and verify that each method and the combined results of the methods are as expected, based on the original definition of the application algorithm.
1. A non-transitory tangible computer readable storage medium having stored thereon a computer program for automated algorithmic verification in an embedded complex distributed storage environment, the computer program including a set of instructions which, when executed by a computer, cause the computer to perform a method comprising the steps of:
creating a model of an algorithm to be tested, the model defining expected execution steps of the algorithm, and expected data structures and process states of the algorithm at the expected execution steps, the model being implemented as script configured to be used by an embedded subsystem automation test system to control execution of the algorithm during testing of the algorithm;
inserting stubs into the algorithm to enable execution of the algorithm to be controlled at the expected execution steps defined in the model, the stubs being small pieces of code that are able to be called by the script, but otherwise do not execute in connection with execution of the algorithm;
beginning execution of the algorithm;
using the stubs, by the embedded subsystem automation test system, to pause execution of the algorithm at one of the execution steps defined in the model and determine data structures and process states of the executing algorithm at the paused execution step; and
comparing the data structures and process states of the executing algorithm at the paused execution step with expected data structures and process states of the algorithm at the paused execution step specified by the model to determine if the algorithm has generated expected data structures and process states at the paused execution step.
2. The non-transitory tangible computer readable storage medium of claim 1 , wherein the model of the algorithm to be tested is based on steps that the algorithm uses to implement particular functions.
3. The non-transitory tangible computer readable storage medium of claim 1 , wherein communication between the embedded subsystem automation test system and stubs is implemented using system calls.
4. The non-transitory tangible computer readable storage medium of claim 1 , further comprising manipulating the data structures of the algorithm by the embedded subsystem automation test system.
5. The non-transitory tangible computer readable storage medium of claim 4 , wherein manipulating data structures comprises:
determining a condition of a first of the data structures to change;
setting a trigger condition for changing the determined condition;
allowing the algorithm to execute;
detecting compliance with the trigger condition; and
modifying the first data structure upon detecting compliance with the trigger condition.
6. The non-transitory tangible computer readable storage medium of claim 1 , wherein the step of comparing the data structures and process states of the executing algorithm at the paused execution step with expected data structures and process states of the algorithm at the paused execution step specified by the model, comprises:
identifying data structure and process states to be verified;
collecting data associated with execution of the algorithm;
preparing the collected data to be consumable by the embedded subsystem automation test system; and
notifying the embedded subsystem automation test system that the collected data is available.
7. The non-transitory tangible computer readable storage medium of claim 1 , further comprising determining, from the step of comparing the data structures and process states, that there is an error in the model of the algorithm.
8. The non-transitory tangible computer readable storage medium of claim 1 , further comprising determining, from the step of comparing the data structures and process states, that there is an error in the algorithm.
9. A method of automated algorithmic verification in an embedded complex distributed storage environment, comprising the steps of:
creating a model of an algorithm to be tested, the model defining expected execution steps of the algorithm, and expected data structures and process states of the algorithm at the expected execution steps, the model being implemented as script configured to be used by an embedded subsystem automation test system to control execution of the algorithm during testing of the algorithm;
inserting stubs into the algorithm to enable execution of the algorithm to be controlled at the expected execution steps defined in the model, the stubs being small pieces of code that are able to be called by the script, but otherwise do not execute in connection with execution of the algorithm;
beginning execution of the algorithm;
using the stubs, by the embedded subsystem automation test system, to pause execution of the algorithm at one of the execution steps defined in the model and determine data structures and process states of the executing algorithm at the paused execution step; and
comparing the data structures and process states of the executing algorithm at the paused execution step with expected data structures and process states of the algorithm at the paused execution step specified by the model to determine if the algorithm has generated expected data structures and process states at the paused execution step.
10. The method of claim 9 , wherein the model of the algorithm to be tested is based on steps that the algorithm uses to implement particular functions.
11. The method of claim 9 , wherein communication between the embedded subsystem automation test system and stubs is implemented using system calls.
12. The method of claim 9 , further comprising manipulating the data structures of the algorithm by the embedded subsystem automation test system.
13. The method of claim 12 , wherein manipulating data structures comprises:
determining a condition of a first of the data structures to change;
setting a trigger condition for changing the determined condition;
allowing the algorithm to execute;
detecting compliance with the trigger condition; and
modifying the first data structure upon detecting compliance with the trigger condition.
14. The method of claim 9 , wherein the step of comparing the data structures and process states of the executing algorithm at the paused execution step with expected data structures and process states of the algorithm at the paused execution step specified by the model, comprises:
identifying data structure and process states to be verified;
collecting data associated with execution of the algorithm;
preparing the collected data to be consumable by the embedded subsystem automation test system; and
notifying the embedded subsystem automation test system that the collected data is available.
15. The method of claim 9 , further comprising determining, from the step of comparing the data structures and process states, that there is an error in the model of the algorithm.
16. The method of claim 9 , further comprising determining, from the step of comparing the data structures and process states, that there is an error in the algorithm.