Persistence points based coverage mechanism for flow testing in high-performance storage systems
A processing device maintains a data structure for a set of process flows executing on one or more processing cores of a storage system. The data structure comprises entries identifying particular execution instances of one or more persistence point functions, each modifying persistent state, with the execution instances having been designated as tested in conjunction with testing of one or more process flows of the set of process flows. The processing device detects an execution instance of a persistence point function of a given one of the process flows, determines whether or not the detected execution instance of the persistence point function has a corresponding entry in the data structure, and responsive to the detected execution instance of the persistence point function not having a corresponding entry in the data structure, initiates a test of the given process flow that includes the detected execution instance of the persistence point function.
1. An apparatus comprising:
at least one processing device comprising a processor coupled to a memory;
said at least processing device being configured:
to maintain a data structure for a set of process flows executing on one or more processing cores of a storage system, the data structure comprising entries identifying particular execution instances of one or more persistence point functions, the execution instances having been designated as tested in conjunction with testing of one or more process flows of the set of process flows, each such persistence point function performing at least one operation that modifies persistent storage of the storage system;
to detect an execution instance of a persistence point function of a given one of the process flows;
to determine whether or not the detected execution instance of the persistence point function has a corresponding entry in the data structure; and
responsive to the detected execution instance of the persistence point function not having a corresponding entry in the data structure, to initiate a test of the given process flow that includes the detected execution instance of the persistence point function.
2. The apparatus of claim 1 wherein said at least one processing device comprises at least a subset of the one or more processing cores of the storage system.
3. The apparatus of claim 1 wherein said at least one processing device is further configured, responsive to the detected execution instance of the persistence point function having a corresponding entry in the data structure, to forego initiation of a test of the given process flow that includes the detected execution instance of the persistence point function.
4. The apparatus of claim 1 wherein the entries of the data structure for respective execution instances of one or more persistence point functions comprise respective call stack identifiers of those execution instances.
5. The apparatus of claim 4 wherein a given one of the call stack identifiers uniquely identifies a particular call stack that ends with a persistence point function as part of its corresponding execution instance of that persistence point function.
6. The apparatus of claim 4 wherein a given one of the call stack identifiers comprises a set of call stack addresses that ends with a call stack address of a persistence point function as part of its corresponding execution instance of that persistence point function.
7. The apparatus of claim 1 wherein a first one of the process flows has a relatively high usage frequency and a second one of the process flows has a relatively low usage frequency.
8. The apparatus of claim 7 wherein the maintaining, detecting, determining and initiating are configured to ensure that the testing of the set of process flows includes at least one test of each of the first and second process flows within a designated time period notwithstanding their distinct usage frequencies.
9. The apparatus of claim 1 wherein initiating a test of the given process flow that includes the detected execution instance of the persistence point function comprises:
initiating an error in the storage system during execution of the given process flow that includes the detected execution instance of the persistence point function; and
monitoring a response of the storage system to the initiated error.
10. The apparatus of claim 9 wherein initiating an error in the storage system during execution of the given process flow comprises at least one of:
injecting an error into the given process flow;
crashing the given process flow; and
simulating a failure of a designated component of the storage system.
11. The apparatus of claim 1 wherein said at least one processing device is further configured to update the data structure maintained for the set of process flows, in conjunction with initiation of the test of the given process flow that includes the detected execution instance of the persistence point function, to comprise an entry identifying the detected execution instance of the persistence point function.
12. The apparatus of claim 1 wherein said at least one processing device is further configured:
to determine a limit on a total number of tests that can be applied within a designated time period in testing the set of process flows; and
to control initiation of tests for detected execution instances of respective persistence point functions not having corresponding entries in the data structure, in accordance with the determined limit.
13. The apparatus of claim 1 wherein the data structure is implemented at least in part utilizing a Bloom filter.
14. A method comprising:
maintaining a data structure for a set of process flows executing on one or more processing cores of a storage system, the data structure comprising entries identifying particular execution instances of one or more persistence point functions, the execution instances having been designated as tested in conjunction with testing of one or more process flows of the set of process flows, each such persistence point function performing at least one operation that modifies persistent storage of the storage system;
detecting an execution instance of a persistence point function of a given one of the process flows;
determining whether or not the detected execution instance of the persistence point function has a corresponding entry in the data structure; and
responsive to the detected execution instance of the persistence point function not having a corresponding entry in the data structure, initiating a test of the given process flow that includes the detected execution instance of the persistence point function;
wherein the method is performed by at least one processing device comprising a processor coupled to a memory.
15. The method of claim 14 wherein the entries of the data structure for respective execution instances of one or more persistence point functions comprise respective call stack identifiers of those execution instances.
16. The method of claim 14 wherein a first one of the process flows has a relatively high usage frequency and a second one of the process flows has a relatively low usage frequency, and wherein the maintaining, detecting, determining and initiating are configured to ensure that the testing of the set of process flows includes at least one test of each of the first and second process flows within a designated time period notwithstanding their distinct usage frequencies.
17. The method of claim 14 further comprising updating the data structure maintained for the set of process flows, in conjunction with initiation of the test of the given process flow that includes the detected execution instance of the persistence point function, to comprise an entry identifying the detected execution instance of the persistence point function.
18. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device comprising a processor coupled to a memory, causes said at least one processing device:
to maintain a data structure for a set of process flows executing on one or more processing cores of a storage system, the data structure comprising entries identifying particular execution instances of one or more persistence point functions, the execution instances having been designated as tested in conjunction with testing of one or more process flows of the set of process flows, each such persistence point function performing at least one operation that modifies persistent storage of the storage system;
to detect an execution instance of a persistence point function of a given one of the process flows;
to determine whether or not the detected execution instance of the persistence point function has a corresponding entry in the data structure; and
responsive to the detected execution instance of the persistence point function not having a corresponding entry in the data structure, to initiate a test of the given process flow that includes the detected execution instance of the persistence point function.
19. The computer program product of claim 18 wherein the entries of the data structure for respective execution instances of one or more persistence point functions comprise respective call stack identifiers of those execution instances.
20. The computer program product of claim 18 wherein a first one of the process flows has a relatively high usage frequency and a second one of the process flows has a relatively low usage frequency, and wherein the maintaining, detecting, determining and initiating are configured to ensure that the testing of the set of process flows includes at least one test of each of the first and second process flows within a designated time period notwithstanding their distinct usage frequencies.