Hiccup-less failback and journal recovery in an active-active storage system
Techniques for providing hiccup-less failback and journal recovery. The techniques include determining a head position of a data log of a primary node and placing a marker at a corresponding position in a data log of a secondary node. In response to an IO request, the head of the primary log is moved to allocate space for writing a data entry, and the secondary node is directed to move a head of the secondary log to allocate space for writing the data entry. In response to a data entry being flushed from the primary node, a tail of the primary log is moved to reclaim allocated space for a previous data entry, and the secondary node is directed to move a tail of the secondary log to reclaim previously allocated space. In response to a tail of the secondary log coinciding with the marker position, content of the logs is synchronized.
1. A method of providing hiccup-less failback and journal recovery in a storage cluster, the storage cluster including at least a primary node and a failback node, each of the primary and failback nodes including a data log in persistent random-access memory (RAM), the method comprising:
in a failback process, placing a marker at a location in the data log of the failback node corresponding to a position of a head pointer of the data log of the primary node;
in response to at least one data entry being flushed from the primary node, moving a tail pointer of the data log of the failback node to reclaim allocated space for at least one corresponding data entry; and
determining that a position of the tail pointer of the data log of the failback node coincides with the location of the marker, thereby indicating that the data log of the failback node is synchronized with the data log of the primary node.
2. The method of claim 1 further comprising:
in response to a first write input/output (IO) request being received at the primary node, moving the head pointer of the data log of the primary node to allocate space for writing at least one first data entry.
3. The method of claim 2 wherein a position of a head pointer of the data log of the failback node coincides with the location of the marker, and wherein the method further comprises:
moving the head pointer of the data log of the failback node to allocate corresponding space for writing a copy of the first data entry.
4. The method of claim 3 further comprising:
writing the at least one first data entry to the allocated space in the data log of the primary node.
5. The method of claim 4 further comprising:
writing the copy of the first data entry to the allocated space in the data log of the failback node.
6. The method of claim 1 further comprising:
prior to determining that the data log of the failback node is synchronized with the data log of the primary node, deferring servicing storage IO requests at the failback node.
7. The method of claim 6 further comprising:
having determined that the data log of the failback node is synchronized with the data log of the primary node, resuming the servicing of the storage IO requests at the fallback node.
8. A system for providing hiccup-less failback and journal recovery in a storage cluster, comprising:
a primary node including a journal in volatile random-access memory (RAM), and a data log in persistent RAM; and
a failback node including a journal in volatile random-access memory (RAM), and a data log in persistent RAM,
wherein, in a failback process, the primary node is configured to execute program instructions to:
place a marker at a location in the data log of the failback node corresponding to a position of a head pointer of the data log of the primary node;
in response to at least one data entry being flushed from the journal of the primary node, direct the failback node to move a tail pointer of the data log of the failback node to reclaim allocated space for at least one corresponding data entry; and
determine that a position of the tail pointer of the data log of the failback node coincides with the location of the marker, thereby indicating that the data log of the failback node is synchronized with the data log of the primary node.
9. The system of claim 8 wherein the primary node is further configured to execute the program instructions, in response to a first write input/output (IO) request being received at the primary node, to move the head pointer of the data log of the primary node to allocate space for writing at least one first data entry.
10. The system of claim 9 wherein a position of a head pointer of the data log of the failback node coincides with the location of the marker, and wherein the primary node is further configured to execute the program instructions to direct the failback node to move the head pointer of the data log of the failback node to allocate corresponding space for writing a copy of the first data entry.
11. The system of claim 10 wherein the primary node is further configured to execute the program instructions to write the at least one first data entry to the allocated space in the data log of the primary node.
12. The system of claim 11 wherein the primary node is further configured to execute the program instructions to direct the failback node to write the copy of the first data entry to the allocated space in the data log of the failback node.
13. The system of claim 8 wherein the failback node is configured to execute program instructions, prior to a determination that the data log of the failback node is synchronized with the data log of the primary node, to defer servicing storage IO requests at the failback node.
14. The system of claim 13 wherein the failback node is further configured to execute the program instructions, in response to the determination that the data log of the failback node is synchronized with the data log of the primary node, to resume the servicing of the storage IO requests at the failback node.
15. The system of claim 8 wherein the data log of each of the primary node and the failback node is configured as a ring buffer.
16. A computer program product including a set of non-transitory, computer-readable media having instructions that, when executed by processing circuitry, cause the processing circuitry to perform a method comprising:
in a failback process of a storage cluster including at least a primary node and a failback node, placing a marker at a location in a data log of the failback node corresponding to a position of a head pointer of a data log of the primary node;
in response to at least one data entry being flushed from the primary node, moving a tail pointer of the data log of the failback node to reclaim allocated space for at least one corresponding data entry; and
determining that a position of the tail pointer of the data log of the failback node coincides with the location of the marker, thereby indicating that the data log of the failback node is synchronized with the data log of the primary node.
17. The computer program product of claim 16 wherein the method further comprises:
in response to a first write input/output (IO) request being received at the primary node, moving the head pointer of the data log of the primary node to allocate space for writing at least one first data entry.
18. The computer program product of claim 17 wherein a position of a head pointer of the data log of the failback node coincides with the location of the marker, and wherein the method further comprises:
moving the head pointer of the data log of the failback node to allocate corresponding space for writing a copy of the first data entry.
19. The computer program product of claim 18 wherein the method further comprises:
writing the at least one first data entry to the allocated space in the data log of the primary node; and
writing the copy of the first data entry to the allocated space in the data log of the failback node.
20. The computer program product of claim 16 wherein the method further comprises:
prior to determining that the data log of the failback node is synchronized with the data log of the primary node, deferring servicing storage IO requests at the failback node; and
having determined that the data log of the failback node is synchronized with the data log of the primary node, resuming the servicing of the storage IO requests at the failback node.