Centralized high-availability flows execution framework
Techniques for providing a framework for handling execution of HA flows in an active-active storage node configuration. The techniques include receiving notifications of functional statuses of processes and/or equipment associated with storage nodes in the active-active configuration, making determinations regarding how to address HA events occurring on the processes and/or equipment associated with the storage nodes based on the received notifications, and, in response to a request to execute an HA flow for a respective HA event, determining whether to refuse the request to execute the HA flow, service the request to execute the HA flow, abort one or more HA flows in execution, and/or postpone execution of the HA flow to a later time based on one or more dependencies defining conditions for the HA flow. In this way, mutual interference of HA flows or other process threads in the active-active configuration can be reduced or eliminated.
1. A method of handling execution of high-availability (HA) process threads in an active-active storage node configuration, comprising:
receiving notifications of functional statuses of processes or equipment associated with active storage nodes in an active-active configuration, the active storage nodes including at least a first active node and a second active node;
determining that a first HA event and a second HA event have occurred on the first active node and the second active node, respectively, in the active-active configuration based on the received notifications;
defining a first HA process thread configured to address the first HA event occurring on the first active node, a second HA process thread being defined and configured to address the second HA event occurring on the second active node;
representing the first HA process thread and the second HA process thread by a first HA process thread object and a second HA process thread object, respectively, each of the first HA process thread object and the second HA process thread object including a first field indicating an HA process thread purpose and a second field indicating an HA process thread execution state, the first HA process thread being defined by a set of parameters including at least one dependency specifying a relationship between the first HA process thread and the second HA process thread as represented by the first HA process thread object and the second HA process thread object, respectively; and
in response to a request to execute the first HA process thread to address the first HA event, performing one of servicing the request to execute the first HA process thread, and postponing execution of the first HA process thread, based on the at least one dependency specifying the relationship between the first HA process thread and the second HA process thread as represented by the first HA process thread object and the second HA process thread object, respectively.
2. The method of claim 1 further comprising:
specifying a set of executable steps for the first HA process thread, the set of parameters further including an abort policy specifying rules regarding whether or when to abort at least the second HA process thread in execution.
3. The method of claim 2 wherein the second field of the second HA process thread object indicates that the HA process thread execution state of the second HA process thread is a state of being in execution, and wherein the method further comprises:
checking the specified rules in the abort policy; and
aborting the second HA process thread in execution based on the specified rules.
4. The method of claim 1 further comprising:
checking the at least one dependency specifying the relationship between the first HA process thread and the second HA process thread.
5. The method of claim 4 further comprising:
in response to the at least one dependency dictating an order in which the first HA process thread and the second HA process thread are to be executed, performing the postponing of the execution of the first HA process thread to satisfy the at least one dependency.
6. The method of claim 2 further comprising:
checking the specified rules in the abort policy; and
aborting all HA process threads in execution based on the specified rules.
7. The method of claim 1 wherein servicing the request to execute the first HA process thread includes initiating execution of the first HA process thread.
8. The method of claim 1 further comprising, in response to the request to execute the first HA process thread to address the first HA event:
allocating the first HA process thread object representing the first HA process thread; and
adding the first HA process thread object to a persistent database, the first HA process thread object being one among a plurality of HA process thread objects added to the persistent database, each of the plurality of HA process thread objects being allocated to represent a respective HA process thread.
9. The method of claim 8 further comprising:
allocating the second HA process thread object representing the second HA process thread; and
adding the second HA process thread object to the persistent database, the second HA process thread object being one among the plurality of HA process thread objects added to the persistent database.
10. The method of claim 1 wherein each of the first HA process thread object and the second HA process thread object further includes a third field indicating an HA process thread execution progress and a fourth field indicating an HA process thread execution result.
11. The method of claim 1 wherein the first field indicates the HA process thread purpose including one of establishing a connection, resetting a disk, and rebooting an active storage node from among the active storage nodes in the active-active configuration.
12. The method of claim 1 wherein the second field indicates the HA process thread execution state including one of an uninitialized state, a state of waiting to be executed, a state of being in execution, and a state of having completed execution.
13. A system for handling execution of high-availability (HA) process threads in an active-active storage node configuration, comprising:
a memory; and
processing circuitry configured to execute program instructions out of the memory to:
receive notifications of functional statuses of processes or equipment associated with active storage nodes in an active-active configuration, the active storage nodes including at least a first active node and a second active node;
determine that a first HA event and a second HA event have occurred on the first active node and the second active node, respectively, in the active-active configuration based on the received notifications
define a first HA process thread configured to address the first HA event occurring on the first active node, a second HA process thread being defined and configured to address the second HA event occurring on the second active node;
represent the first HA process thread and the second HA process thread by a first HA process thread object and a second HA process thread object, respectively, each of the first HA process thread object and the second HA process thread object including a first field indicating an HA process thread purpose and a second field indicating an HA process thread execution state, the first HA process thread being defined by a set of parameters including at least one dependency specifying a relationship between the first HA process thread and the second HA process thread as represented by the first HA process thread object and the second HA process thread object, respectively; and
in response to a request to execute the first HA process thread to address the first HA event, perform one of servicing the request to execute the first HA process thread, and postponing execution of the first HA process thread, based on the at least one dependency specifying the relationship between the first HA process thread and the second HA process thread as represented by the first HA process thread object and the second HA process thread object, respectively.
14. The system of claim 13 wherein the processing circuitry is further configured to execute the program instructions out of the memory to specify a set of executable steps for the first HA process thread, wherein the set of parameters further includes an abort policy specifying rules regarding whether or when to abort at least the second HA process thread in execution.
15. The system of claim 14 wherein the second field of the second HA process thread object indicates that the HA process thread execution state of the second HA process thread is a state of being in execution, and wherein the processing circuitry is further configured to execute the program instructions out of the memory to check the specified rules in the abort policy and abort the second HA process thread in execution based on the specified rules.
16. The system of claim 13 wherein the processing circuitry is further configured to execute the program instructions out of the memory to check the at least one dependency specifying the relationship between the first HA process thread and the second HA process thread.
17. The system of claim 16 wherein the processing circuitry is further configured to execute the program instructions out of the memory, in response to the at least one dependency dictating an order in which the first HA process thread and the other second HA process thread are to be executed, to perform the postponing of the execution of the first HA process thread to satisfy the at least one dependency.
18. The system of claim 14 wherein the processing circuitry is further configured to execute the program instructions out of the memory to:
check the specified rules in the abort policy; and
abort all HA process threads in execution based on the specified rules.
19. The system of claim 13 wherein servicing the request to execute the first HA process thread includes initiating execution of the first HA process thread.
20. A computer program product including a set of non-transitory, computer-readable media having instructions that, when executed by processing circuitry, cause the processing circuitry to perform a method of handling execution of high-availability (HA) process threads in an active-active storage node configuration, the method comprising
receiving notifications of functional statuses of processes or equipment associated with active storage nodes in an active-active configuration, the active storage nodes including at least a first active node and a second active node;
determining that a first HA event and a second HA event have occurred on the first active node and the second active node, respectively, in the active-active configuration based on the received notifications;
defining a first HA process thread configured to address the first HA event occurring on the first active node, a second HA process thread being defined and configured to address the second HA event occurring on the second active node;
representing the first HA process thread and the second HA process thread by a first HA process thread object and a second HA process thread object, respectively, each of the first HA process thread object and the second HA process thread object including a first field indicating an HA process thread purpose and a second field indicating an HA process thread execution state, the first HA process thread being defined by a set of parameters including at least one dependency specifying a relationship between the first HA process thread and the second HA process thread as represented by the first HA process thread object and the second HA process thread object, respectively; and
in response to a request to execute the first HA process thread to address the first HA event, performing one of servicing the request to execute the first HA process thread, and postponing execution of the first HA process thread, based on the at least one dependency specifying the relationship between the first HA process thread and the second HA process thread as represented by the first HA process thread object and the second HA process thread object, respectively.