Techniques for replication checkpointing during disaster recovery
Techniques are described for checkpointing multiple key ranges in parallel and concurrently during file storage replications between file systems in different cloud infrastructure regions. In certain embodiments, multiple range threads processing multiple key ranges, one thread per key range, create checkpoints for their respective key ranges in parallel and concurrently after processing a per-determined number of B-tree keys. In certain embodiments, each thread requests a lock from a central checkpoint record and takes turns for updating a status byte while continuing processing the B-tree keys in its responsible key range. In certain embodiments, upon encountering a failure event, either a system crash or a thread failure, each thread restarts its B-tree key processing from a B-tree key after the most recent checkpoint.
1. A method, comprising:
receiving, by a computing system, a request for a file system replication between a source file system and a target file system, the source file system and the target file system being in different regions;
creating, by a first processing thread of the computing system, a first checkpoint in a first key range after fulfilling a predetermined requirement for processing first binary tree (B-tree) key-value pairs in the first key range;
requesting, by the first processing thread of the computing system, a lock to update a central record after creating the first checkpoint in the first key range;
creating, by a second processing thread of the computing system, a second checkpoint in a second key range after fulfilling the predetermined requirement for processing second B-tree key-value pairs in the second key range, wherein the processing the first B-tree key-value pairs in the first key range by the first processing thread is performed in parallel with the processing the second B-tree key-value pairs in the second key range by the second processing thread;
requesting, by the second processing thread of the computing system, the lock to update the central record after creating the second checkpoint in the second key range; and
granting, by the computing system, the lock to the first processing thread to update the central record, the request by the first processing thread being earlier than the request by the second processing thread, the lock being configured to prevent the second process thread from updating the central record.
2. The method of claim 1 , further comprising:
processing, by the first processing thread, more B-tree key-value pairs in the first key range after updating the central record; and
granting, by the computing system, the lock to the second processing thread to update the central record after the first processing thread completes updating the central record.
3. The method of claim 1 , further comprising processing, by the second processing thread, more B-tree key-value pairs in the second key range while waiting for the lock.
4. The method of claim 1 , wherein the first processing thread operates independently from the second processing thread.
5. The method of claim 1 , further comprising restarting processing the first B-tree key-value pairs in the first key range by the first processing thread from a B-tree key after the first checkpoint in the first key range upon detecting a system failure.
6. The method of claim 1 , further comprising restarting processing the first B-tree key-value pairs in the first key range by a third processing thread from a B-tree key after the first checkpoint in the first key range upon detecting a failure of the first processing thread.
7. A non-transitory computer-readable medium storing computer-executable instructions that, when executed by one or more processors of a computing system, cause the one or more processors to perform operations comprising:
receiving, by the computing system, a request for a file system replication between a source file system and a target file system, the source file system and the target file system being in different regions;
creating, by a first processing thread of the computing system, a first checkpoint in a first key range after fulfilling a predetermined requirement for processing first binary tree (B-tree) key-value pairs in the first key range;
requesting, by the first processing thread of the computing system, a lock to update a central record after creating the first checkpoint in the first key range;
creating, by a second processing thread of the computing system, a second checkpoint in a second key range after fulfilling a predetermined requirement for processing second B-tree key-value pairs in the second key range, wherein the processing the first B-tree key-value pairs in the first key range by the first processing thread is performed in parallel with the processing the second B-tree key-value pairs in the second key range by the second processing thread;
requesting, by the second processing thread of the computing system, the lock to update the central record after creating the second checkpoint in the second key range; and
granting, by the computing system, the lock to the first processing thread to update the central record, the request by the first processing thread being earlier than the request by the second processing thread, the lock being configured to prevent the second process thread from updating the central record.
8. The non-transitory computer-readable medium of claim 7 , the operations further comprising:
processing, by the first processing thread, more B-tree key-value pairs in the first key range after updating the central record; and
granting, by the computing system, the lock to the second processing thread to update the central record after the first processing thread completes updating the central record.
9. The non-transitory computer-readable medium of claim 7 , wherein the first processing thread operates independently from the second processing thread.
10. The non-transitory computer-readable medium of claim 7 , the operations further comprising restarting processing the first B-tree key-value pairs in the first key range by the first processing thread from a B-tree key after the first checkpoint in the first key range upon detecting a system failure.
11. The non-transitory computer-readable medium of claim 7 , the operations further comprising restarting processing the first B-tree key-value pairs in the first key range by a third processing thread from a B-tree key after the first checkpoint in the first key range upon detecting a failure of the first processing thread.
12. A system, comprising:
one or more processors; and
one or more computer readable media storing computer-executable instructions that, when executed by the one or more processors, cause the system to:
receive a request for a file system replication between a source file system and a target file system, the source file system and the target file system being in different regions;
create, by a first processing thread of the system, a first checkpoint in a first key range after fulfilling a predetermined requirement for processing first binary tree (B-tree) key-value pairs in the first key range;
request, by the first processing thread of the system, a lock to update a central record after creating the first checkpoint in the first key range;
create, by a second processing thread of the system, a second checkpoint in a second key range after fulfilling a predetermined requirement for processing second B-tree key-value pairs in the second key range, wherein the processing the first B-tree key-value pairs in the first key range by the first processing thread is performed in parallel with the processing the second B-tree key-value pairs in the second key range by the second processing thread;
request, by the second processing thread, the lock to update the central record after creating the second checkpoint in the second key range; and
grant the lock to the first processing thread to update the central record, the request by the first processing thread being earlier than the request by the second processing thread, the lock being configured to prevent the second process thread from updating the central record.
13. The system of claim 12 , wherein the system is further caused to:
process, by the first processing thread of the system, more B-tree key-value pairs in the first key range after updating the central record; and
grant the lock to the second processing thread to update the central record after the first processing thread completes updating the central record.
14. The system of claim 12 , wherein the first processing thread operates independently from the second processing thread.
15. The system of claim 12 , wherein the system is further caused to restart processing the first B-tree key-value pairs in the first key range by the first processing thread from a B-tree key after the first checkpoint in the first key range upon detecting a system failure.
16. The system of claim 12 , wherein the system is further caused to restart processing the first B-tree key-value pairs in the first key range by a third processing thread from a B-tree key after the first checkpoint in the first key range upon detecting a failure of the first processing thread.