Task-based framework for synchronization of event handling between nodes in an active/active data storage system
Technology for synchronizing event handling between a primary storage processor and a secondary storage processor in an active/active data storage system. A single working task descriptor is selected at a time for uninterruptible execution, from a task scheduling queue that is maintained only on the primary storage processor. A copy of the working task descriptor is transmitted to the secondary storage processor. The working task descriptor is executed on both the primary storage processor and the secondary storage processor and without interruption by execution of any other task descriptor. Execution of the working task descriptor includes simultaneously transitioning through individual task states on both the primary storage processor and the secondary storage processor, and simultaneously executing functions on both the primary storage processor and the secondary storage processor during associated task states.
1. A method of synchronizing event handling between a primary storage processor and a secondary storage processor in an active/active data storage system, comprising:
maintaining a task scheduling queue only on the primary storage processor, wherein the task scheduling queue contains a plurality of task descriptors, wherein each task descriptor is associated with an event and is placed on the scheduling queue in response to detection of the associated event, wherein each task descriptor indicates an ordered set of task states, wherein each one of the task states is associated with a respective one of a plurality of functions that are to be executed in response to detection of the associated event;
selecting, from the task scheduling queue on the primary storage processor, a single working task descriptor at a time for execution on both the primary storage processor and the secondary storage processor;
sending a copy of the working task descriptor from the primary storage processor to the secondary storage processor; and
executing the working task descriptor on both the primary storage processor and the secondary storage processor, wherein execution of the working task descriptor is uninterruptable by execution of any other task descriptor, and wherein executing the working task descriptor includes simultaneously transitioning through individual ones of the task states indicated by the working task descriptor on both the primary storage processor and the secondary storage processor, one task state at a time, in the order of the task states indicated in the working task descriptor, and, during each one of the task states indicated in the working task descriptor, executing the function associated with the task state on both the primary storage processor and the secondary storage processor.
2. The method of claim 1 , wherein simultaneously transitioning through each individual one of the task states indicated by the working task descriptor on both the primary storage processor and the secondary storage processor, under non-failure conditions, comprises simultaneously transitioning through a synching stage, a started stage, and a done stage on both the primary storage processor and the secondary storage processor for each individual one of the task states indicated by the working task descriptor.
3. The method of claim 2 , wherein the primary storage processor and the secondary storage processor simultaneously transition through the synching stage, the started stage, and the done stage for each individual one of the task states indicated by the working task descriptor using a synching cluster flag and a done cluster flag on the primary storage processor and a synching cluster flag and a done cluster flag on the secondary storage processor.
4. The method of claim 3 , wherein executing the function associated with the task state on both the primary storage processor and the secondary storage processor during each one of the task states indicated in the working task descriptor comprises executing the function associated with the task state during the started stage on both the primary storage processor and the secondary storage processor.
5. The method of claim 1 , further comprising
wherein each task descriptor contained in the task scheduling queue further indicates a rollback function to be executed on both the primary storage processor and the secondary storage processor in response to detecting a failure while executing the task descriptor; and
in response to detecting a failure while executing a function associated with one of the task states indicated by the working task descriptor, executing a rollback function indicated by the working task descriptor on both the primary storage processor and the secondary storage processor.
6. The method of claim 5 , further comprising:
wherein each task descriptor contained in the task scheduling queue further indicates a peer panic function to be executed in response to detecting a failure of either the primary storage processor or the secondary storage processor; and
in response to detecting a failure of the primary storage processor while executing the working task descriptor, executing a peer panic function indicated by the working task descriptor on the secondary storage processor.
7. The method of claim 5 , further comprising:
wherein each task descriptor contained in the task scheduling queue further indicates a peer panic function to be executed in response to detecting a failure of either the primary storage processor or the secondary storage processor; and
in response to detecting a failure of the secondary storage processor while executing the working task descriptor, executing a peer panic function indicated by the working task descriptor on the primary storage processor.
8. The method of claim 1 , further comprising:
in response to detecting an event on the secondary storage processor, sending a message from the secondary storage processor to the primary storage processor that contains a task descriptor associated with the event detected on the secondary storage processor;
in response to receipt by the primary storage processor of the message that contains the task descriptor associated with the event detected on the secondary storage processor, adding, by the primary storage processor, the task descriptor associated with the event detected on the secondary storage processor to the task scheduling queue maintained only on the primary storage processor; and
subsequently selecting the task descriptor associated with the event detected on the secondary storage processor from the task scheduling queue as a working task descriptor for execution on both the primary storage processor and the secondary storage processor.
9. The method of claim 1 , wherein both the primary storage processor and the secondary storage processor receive and process host write I/O operations directed to a storage object; and
wherein both the primary storage processor and the secondary storage processor store host data written to the storage object into non-volatile storage located on one or more non-volatile data storage drives.
10. The method of claim 1 , wherein both the primary storage processor and the secondary storage processor receive and process host write I/O operations directed to a storage object; and
wherein both the primary storage processor and the secondary storage processor store host data written to the storage object into at least one persistent cache.
11. The method of claim 1 , wherein host data written to the storage object by host write I/O operations received and processed by the primary storage processor is stored by the primary storage processor in a primary copy of the storage object that is maintained by the primary storage processor, and is also mirrored to the secondary storage processor for the secondary storage processor to store in a secondary copy of the storage object that is maintained by the secondary storage processor; and
wherein host data written to the storage object by host write I/O operations received and processed by the secondary storage processor is stored by the secondary storage processor in the secondary copy of the storage object that is maintained by the secondary storage processor, and is also mirrored to the primary storage processor for the primary storage processor to store in the primary copy of the storage object that is maintained by the primary storage processor.
12. An active/active data storage system, comprising:
a primary storage processor including processing circuitry and a memory having program code stored thereon;
a secondary storage processor including processing circuitry and a memory having program code stored thereon;
wherein the program code stored in the memory of the primary storage processor and the memory of the secondary storage processor, when executed by the processing circuitry in the primary storage processor and the processing circuitry in the secondary storage processor, causes the processing circuitry in the primary storage processor and the processing circuitry in the secondary storage processor to:
maintain a task scheduling queue only on the primary storage processor, wherein the task scheduling queue contains a plurality of task descriptors, wherein each task descriptor is associated with an event and is placed on the scheduling queue in response to detection of the associated event, wherein each task descriptor indicates an ordered set of task states, wherein each one of the task states is associated with a respective one of a plurality of functions that are to be executed in response to detection of the associated event,
select, from the task scheduling queue on the primary storage processor, a single working task descriptor at a time for execution on both the primary storage processor and the secondary storage processor,
send a copy of the working task descriptor from the primary storage processor to the secondary storage processor, and
execute the working task descriptor on both the primary storage processor and the secondary storage processor, wherein execution of the working task descriptor is uninterruptable by execution of any other task descriptor, and wherein executing the working task descriptor includes simultaneously transitioning through individual ones of the task states indicated by the working task descriptor on both the primary storage processor and the secondary storage processor, one task state at a time, in the order of the task states indicated in the working task descriptor, and, during each one of the task states indicated in the working task descriptor, executing the function associated with the task state on both the primary storage processor and the secondary storage processor.
13. The active/active data storage system of claim 12 , wherein the program code stored in the memory of the primary storage processor and the memory of the secondary storage processor, when executed by the processing circuitry in the primary storage processor and the processing circuitry in the secondary storage processor, further causes the processing circuitry in the primary storage processor and the processing circuitry in the secondary storage processor to simultaneously transition the primary storage processor and the secondary storage processor through each individual one of the task states indicated by the working task descriptor, under non-failure conditions, by causing the primary storage processor and the secondary storage processor to simultaneously transition through a synching stage, a started stage, and a done stage for each individual one of the task states indicated by the working task descriptor.
14. The active/active data storage system of claim 13 , wherein the program code stored in the memory of the primary storage processor and the memory of the secondary storage processor, when executed by the processing circuitry in the primary storage processor and the processing circuitry in the secondary storage processor, further causes the processing circuitry in the primary storage processor and the processing circuitry in the secondary storage processor to simultaneously transition the primary storage processor and the secondary storage processor through the synching stage, the started stage, and the done stage for each individual one of the task states indicated by the working task descriptor using a synching cluster flag and a done cluster flag on the primary storage processor and a synching cluster flag and a done cluster flag on the secondary storage processor.
15. The active/active data storage system of claim 14 , wherein the program code stored in the memory of the primary storage processor and the memory of the secondary storage processor, when executed by the processing circuitry in the primary storage processor and the processing circuitry in the secondary storage processor, further causes the processing circuitry in the primary storage processor and the processing circuitry in the secondary storage processor to execute the function associated with the task state on both the primary storage processor and the secondary storage processor during each one of the task states indicated by the working task descriptor at least in part by executing the function associated with the task state during the started stage on both the primary storage processor and the secondary storage processor.
16. The active/active data storage system of claim 12 , wherein each task descriptor contained in the task scheduling queue further indicates a rollback function to be executed on both the primary storage processor and the secondary storage processor in response to detecting a failure while executing the task descriptor; and
wherein the program code stored in the memory of the primary storage processor and the memory of the secondary storage processor, when executed by the processing circuitry in the primary storage processor and the processing circuitry in the secondary storage processor, further causes the processing circuitry in the primary storage processor and the processing circuitry in the secondary storage processor to, in response to detection of a failure while executing a function associated with one of the task states indicated by the working task descriptor, execute a rollback function indicated by the working task descriptor on both the primary storage processor and the secondary storage processor.
17. The active/active data storage system of claim 16 , wherein each task descriptor contained in the task scheduling queue further indicates a peer panic function to be executed in response to detecting a failure of either the primary storage processor or the secondary storage processor; and
wherein the program code stored in the memory of the primary storage processor and the memory of the secondary storage processor, when executed by the processing circuitry in the primary storage processor and the processing circuitry in the secondary storage processor, further causes the processing circuitry in the primary storage processor and the processing circuitry in the secondary storage processor to, in response to detection of a failure of the primary storage processor while executing the working task descriptor, execute a peer panic function indicated by the working task descriptor on the secondary storage processor.
18. The active/active data storage system of claim 16 , wherein each task descriptor contained in the task scheduling queue further indicates a peer panic function to be executed in response to detecting a failure of either the primary storage processor or the secondary storage processor; and
wherein the program code stored in the memory of the primary storage processor and the memory of the secondary storage processor, when executed by the processing circuitry in the primary storage processor and the processing circuitry in the secondary storage processor, further causes the processing circuitry in the primary storage processor and the processing circuitry in the secondary storage processor to, in response to detection of a failure of the secondary storage processor while executing the working task descriptor, execute a peer panic function indicated by the working task descriptor on the primary storage processor.
19. The active/active data storage system of claim 12 , wherein the program code stored in the memory of the primary storage processor and the memory of the secondary storage processor, when executed by the processing circuitry in the primary storage processor and the processing circuitry in the secondary storage processor, further causes the processing circuitry in the primary storage processor and the processing circuitry in the secondary storage processor to:
in response to detecting an event on the secondary storage processor, send a message from the secondary storage processor to the primary storage processor that contains a task descriptor associated with the event detected on the secondary storage processor;
in response to receipt by the primary storage processor of the message that contains the task descriptor associated with the event detected on the secondary storage processor, add the task descriptor associated with the event detected on the secondary storage processor to the task scheduling queue maintained only on the primary storage processor; and
select the task descriptor associated with the event detected on the secondary storage processor from the task scheduling queue as a working task descriptor for execution on both the primary storage processor and the secondary storage processor.
20. A non-transitory computer readable medium for synchronizing event handling between a primary storage processor and a secondary storage processor in an active/active data storage system, the non-transitory computer readable medium comprising instructions stored thereon that when executed on processing circuitry perform the steps of:
maintaining a task scheduling queue only on the primary storage processor, wherein the task scheduling queue contains a plurality of task descriptors, wherein each task descriptor is associated with an event and is placed on the scheduling queue in response to detection of the associated event, wherein each task descriptor indicates an ordered set of task states, wherein each one of the task states is associated with a respective one of a plurality of functions that are to be executed in response to detection of the associated event;
selecting, from the task scheduling queue on the primary storage processor, a single working task descriptor at a time for execution on both the primary storage processor and the secondary storage processor;
sending a copy of the working task descriptor from the primary storage processor to the secondary storage processor; and
executing the working task descriptor on both the primary storage processor and the secondary storage processor, wherein execution of the working task descriptor is uninterruptable by execution of any other task descriptor, and wherein executing the working task descriptor includes simultaneously transitioning through individual ones of the task states indicated by the working task descriptor on both the primary storage processor and the secondary storage processor, one task state at a time, in the order of the task states indicated in the working task descriptor, and, during each one of the task states indicated in the working task descriptor, executing the function associated with the task state on both the primary storage processor and the secondary storage processor.