Method for seamless failback after unplanned failover
Systems and methods are directed to seamless failback after an unplanned failover. The method involves executing a truncate-and-restore command on a primary topic of a primary cluster, performing checks on a secondary topic to ensure it is a mirror topic in a stopped state with valid identifiers and offsets, and transitioning the primary topic to immutable state. The method further includes comparing sequence numbers to ensure safe truncation, truncating partitions to match the secondary topic's log end offsets, and clamping consumer group offsets. The primary topic is then converted to a mirror state, enabling active mirroring. A reverse command is then executed to complete the failback process, restoring the primary topic to a writable state.
1 . A method for seamless failback after an unplanned failover, the method comprising:
executing a truncate-and-restore command on a primary topic of a primary cluster after the primary cluster becomes operational after the unplanned failover;
responsive to executing the truncate-and-restore command, performing a plurality of checks on a corresponding secondary topic of a secondary cluster that was failed over to during the unplanned failover;
determining stopped log end offsets of the corresponding secondary topic of the secondary cluster;
converting the primary topic of the primary cluster to an immutable state based on successful validation of the plurality of checks;
truncating partitions of the primary topic to log end offsets corresponding to the stopped log end offsets of the corresponding secondary topic;
after the truncating, converting the primary topic to a mirror state and enabling active mirroring of new data written to the corresponding secondary topic after the unplanned failover; and
at or near zero mirror lag, executing a reverse command to switch a direction of mirroring flow and convert the primary topic to a writable state.
2 . The method of claim 1 , wherein the plurality of checks comprises one or more of:
verifying that the corresponding secondary topic is a mirror topic;
verifying that the corresponding secondary topic is in a stopped mirror state;
verifying that a source topic identifier of the corresponding secondary topic matches a local topic identifier of the primary topic; or
confirming that the corresponding secondary topic has valid stopped log end offsets.
3 . The method of claim 1 , further comprising:
comparing a sequence number associated with the primary topic with a sequence number associated with the corresponding secondary topic, wherein the truncating occurs based on the sequence number associated with the primary topic being lower than the sequence number associated with the corresponding secondary topic.
4 . The method of claim 3 , wherein the sequence number is a monotonically increasing integer that increments each time a mirror topic transitions to a stopped state.
5 . The method of claim 1 , further comprising:
clamping consumer group offsets associated with the primary topic to a minimum of a persisted consumer group offset or truncated log end offset.
6 . The method of claim 1 , wherein the reverse command comprises a reverse-and-start command that immediately transitions the corresponding secondary topic to the mirror state after the failback.
7 . The method of claim 1 , wherein the reverse command comprises a reverse-and-pause command that places the corresponding secondary topic in a paused mirror state until a resume-mirror command is executed.
8 . The method of claim 1 , further comprising:
performing a periodic remote call to the corresponding secondary topic to obtain the stopped log end offsets and epochs from the corresponding secondary topic.
9 . The method of claim 1 , further comprising:
writing divergent records to a special internal topic before truncating the primary topic.
10 . A system for seamless failback after an unplanned failover, the system comprising:
one or more hardware processors; and
one or more storage components storing instructions that, when executed by the one or more hardware processors, cause the one or more hardware processors to perform operations comprising:
executing a truncate-and-restore command on a primary topic of a primary cluster after the primary cluster becomes operational after the unplanned failover;
responsive to executing the truncate-and-restore command, performing a plurality of checks on a corresponding secondary topic of a secondary cluster that was failed over to during the unplanned failover;
determining stopped log end offsets of the corresponding secondary topic of the secondary cluster;
converting the primary topic of the primary cluster to an immutable state based on successful validation of the plurality of checks;
truncating partitions of the primary topic to log end offsets corresponding to the stopped log end offsets of the corresponding secondary topic;
after the truncating, converting the primary topic to a mirror state and enabling active mirroring of new data written to the corresponding secondary topic after the unplanned failover; and
at or near zero mirror lag, executing a reverse command to switch a direction of mirroring flow and convert the primary topic to a writable state.
11 . The system of claim 10 , wherein the plurality of checks comprises one or more of:
verifying that the corresponding secondary topic is a mirror topic;
verifying that the corresponding secondary topic is in a stopped mirror state;
verifying that a source topic identifier of the corresponding secondary topic matches a local topic identifier of the primary topic; or
confirming that the corresponding secondary topic has valid stopped log end offsets.
12 . The system of claim 10 , wherein the operations further comprise:
comparing a sequence number associated with the primary topic with a sequence number associated with the corresponding secondary topic, wherein the truncating occurs based on the sequence number associated with the primary topic being lower than the sequence number associated with the corresponding secondary topic.
13 . The system of claim 12 , wherein the sequence number is a monotonically increasing integer that increments each time a mirror topic transitions to a stopped state.
14 . The system of claim 10 , wherein the operations further comprise:
clamping consumer group offsets associated with the primary topic to a minimum of a persisted consumer group offset or truncated log end offset.
15 . The system of claim 10 , wherein the reverse command comprises a reverse-and-start command that immediately transitions the corresponding secondary topic to the mirror state after the failback.
16 . The system of claim 10 , wherein the reverse command comprises a reverse-and-pause command that places the corresponding secondary topic in a paused mirror state until a resume-mirror command is executed.
17 . The system of claim 10 , wherein the operations further comprise:
performing a periodic remote call to the corresponding secondary topic to obtain the stopped log end offsets and epochs from the corresponding secondary topic.
18 . A machine-storage medium comprising instructions which, when executed by one or more hardware processors of a machine, cause the machine to perform operations for seamless failback after an unplanned failover, the operations comprising:
executing a truncate-and-restore command on a primary topic of a primary cluster after the primary cluster becomes operational after the unplanned failover;
responsive to executing the truncate-and-restore command, performing a plurality of checks on a corresponding secondary topic of a secondary cluster that was failed over to during the unplanned failover;
determining stopped log end offsets of the corresponding secondary topic of the secondary cluster;
converting the primary topic of the primary cluster to an immutable state based on successful validation of the plurality of checks;
truncating partitions of the primary topic to log end offsets corresponding to the stopped log end offsets of the corresponding secondary topic;
after the truncating, converting the primary topic to a mirror state and enabling active mirroring of new data written to the corresponding secondary topic after the unplanned failover; and
at or near zero mirror lag, executing a reverse command to switch a direction of mirroring flow and convert the primary topic to a writable state.