Techniques for replication checkpointing during disaster recovery
Techniques are described for checkpointing multiple key ranges in parallel and concurrently during file storage replications between file systems in different cloud infrastructure regions. In certain embodiments, multiple range threads processing multiple key ranges, one thread per key range, create checkpoints for their respective key ranges in parallel and concurrently after processing a per-determined number of B-tree keys. In certain embodiments, upon encountering a failure event, either a system crash or a thread failure, each thread restarts its B-tree key processing from a B-tree key after the most recent checkpoint.
1. A method, comprising:
receiving, by a computing system, a request for a file system replication between a source file system and a target file system, the source file system and the target file system being in different regions; and
in response to the receiving the request:
processing, by a first processing thread of the computing system, first binary tree (B-tree) key-value pairs in a first key range;
creating, by the first processing thread of the computing system, a first checkpoint in a first key range at a first time after fulfilling a requirement for processing the first B-tree key-value pairs in the first key range, wherein the creating the first checkpoint in the first key range comprises updating a first checkpoint record stored in a database and associated with the first key range;
processing, by a second processing thread of the computing system, second B-tree key-value pairs in a second key range, the first processing thread being configured to operate independently from the second processing thread; and
creating, by the second processing thread of the computing system, a second checkpoint in a second key range at a second time after fulfilling the requirement for processing the second B-tree key-value pairs in the second key range, the creation of the first checkpoint in the first key range being configured to be performed in parallel with the creation of the second checkpoint in the second key range, wherein the creating the second checkpoint in the second key range comprises updating a second checkpoint record stored in the database and associated with the second key range.
2. The method of claim 1 , wherein the requirement is a time period or a number of processed B-tree key-value pairs.
3. The method of claim 1 , wherein the first time is different from the second time.
4. The method of claim 1 , further comprising:
restarting the processing of the first B-tree key-value pairs in the first key range by the first processing thread after the first checkpoint in the first key range upon detecting a system failure; and
restarting the processing of the second B-tree key-value pairs in the second key range by the second processing thread after the second checkpoint in the second key range upon detecting the system failure.
5. The method of claim 1 , further comprising restarting the processing of the first B-tree key-value pairs in the first key range by a third processing thread after the first checkpoint in the first key range upon detecting a failure of the first processing thread.
6. The method of claim 5 , wherein restarting the processing of the first B-tree key-value pairs in the first key range by a third processing thread comprises:
taking, by the third processing thread, ownership of a first checkpoint record associated with the first key range from the first processing thread, wherein the ownership comprises a lock for updating the first checkpoint record; and
reusing, by the third processing thread, information in the first checkpoint record associated with the first key range.
7. The method of claim 1 ,
wherein the first checkpoint record associated with the first key range comprises an identification for the first key range, number of the first B-tree key-value pairs processed in the first key range, and a next key to restart in the first key range upon encountering a failure; and
wherein the second checkpoint record associated with the second key range comprises an identification for the second key range, number of the second B-tree key-value pairs processed in the second key range, and a next key to restart in the second key range upon encountering a failure.
8. The method of claim 1 , further comprising updating a central record by the first processing thread for the first key range, and by the second processing thread for the second key range.
9. A non-transitory computer-readable medium storing computer-executable instructions that, when executed by one or more processors of a computing system, cause the one or more processors to perform operations comprising:
receiving, by a computing system, a request for a file system replication between a source file system and a target file system, the source file system and the target file system being in different regions; and
in response to the receiving the request:
processing, by a first processing thread of the computing system, first binary tree (B-tree) key-value pairs in a first key range;
creating, by the first processing thread of the computing system, a first checkpoint in a first key range at a first time after fulfilling a requirement for processing the first B-tree key-value pairs in the first key range, wherein the creating the first checkpoint in the first key range comprises updating a first checkpoint record stored a database and associated with the first key range;
processing, by a second processing thread of the computing system, second B-tree key-value pairs in a second key range, the first processing thread being configured to operate independently from the second processing thread; and
creating, by the second processing thread of the computing system, a second checkpoint in a second key range at a second time after fulfilling the requirement for processing the second B-tree key-value pairs in the second key range, the creation of the first checkpoint in the first key range being configured to perform in parallel with the creation of the second checkpoint in the second key range, wherein the creating the second checkpoint in the second key range comprises updating a second checkpoint record stored in the database and associated with the second key range.
10. The non-transitory computer-readable medium of claim 9 , further comprising:
restarting the processing of the first B-tree key-value pairs in the first key range by the first processing thread after the first checkpoint in the first key range upon detecting a system failure; and
restarting the processing of the second B-tree key-value pairs in the second key range by the second processing thread after the second checkpoint in the second key range upon detecting the system failure.
11. The non-transitory computer-readable medium of claim 9 , further comprising restarting the processing of the first B-tree key-value pairs in the first key range by a third processing thread after the first checkpoint in the first key range upon detecting a failure of the first processing thread.
12. The non-transitory computer-readable medium of claim 11 , wherein restarting the processing of the first B-tree key-value pairs in the first key range by a third processing thread comprises:
taking, by the third processing thread, ownership of a first checkpoint record associated with the first key range from the first processing thread, wherein the ownership comprises a lock for updating the first checkpoint record; and
reusing, by the third processing thread, information in the first checkpoint record associated with the first key range.
13. The non-transitory computer-readable medium of claim 9 ,
wherein the first checkpoint record associated with the first key range comprises an identification for the first key range, number of the first B-tree key-value pairs processed in the first key range, and a next key to restart in the first key range upon encountering a failure; and
wherein the second checkpoint record associated with the second key range comprises an identification for the second key range, number of the second B-tree key-value pairs processed in the second key range, and a next key to restart in the second key range upon encountering a failure.
14. The non-transitory computer-readable medium of claim 9 , further comprising updating a central record by the first processing thread for the first key range, and by the second processing thread for the second key range.
15. The non-transitory computer-readable medium of claim 9 , wherein the requirement is a time period or a number of processed B-tree key-value pairs.
16. A system, comprising:
one or more processors; and
one or more computer readable media storing computer-executable instructions that, when executed by the one or more processors, cause the system to:
receive a request for a file system replication between a source file system and a target file system, the source file system and the target file system being in different regions; and
in response to the receiving the request:
process, by a first processing thread of the system, first binary tree (B-tree) key-value pairs in a first key range;
create, by the first processing thread of the system, a first checkpoint in a first key range at a first time after fulfilling a requirement for processing the first B-tree key-value pairs in the first key range, wherein the creating the first checkpoint in the first key range comprises updating a first checkpoint record stored in a database and associated with the first key range;
process, by a second processing thread of the system, second B-tree key-value pairs in a second key range, the first processing thread being configured to operate independently from the second processing thread; and
create, by the second processing thread of the system, a second checkpoint in a second key range at a second time after fulfilling the requirement for processing the second B-tree key-value pairs in the second key range, the creation of the first checkpoint in the first key range being configured to perform in parallel with the creation of the second checkpoint in the second key range, wherein the creating the second checkpoint in the second key range comprises updating a second checkpoint record stored in the database and associated with the second key range.
17. The system of claim 16 , wherein the system is further caused to:
restart the processing of the first B-tree key-value pairs in the first key range by the first processing thread after the first checkpoint in the first key range upon detecting a system failure; and
restart the processing of the second B-tree key-value pairs in the second key range by the second processing thread after the second checkpoint in the second key range upon detecting the system failure.
18. The system of claim 16 , wherein the system is further caused to:
restart the processing of the first B-tree key-value pairs in the first key range by a third processing thread after the first checkpoint in the first key range upon detecting a failure of the first processing thread.
19. The system of claim 18 , wherein restarting the processing of the first B-tree key-value pairs in the first key range by a third processing thread comprises:
take, by the third processing thread, ownership of a first checkpoint record associated with the first key range from the first processing thread, wherein the ownership comprises a lock for updating the first checkpoint record; and
reuse, by the third processing thread, information in the first checkpoint record associated with the first key range.
20. The system of claim 16 , wherein the system is further caused to update a central record by the first processing thread for the first key range, and by the second processing thread for the second key range.