Efficient mechanism for data protection against cloud region failure or site disasters and recovery time objective (RTO) improvement for backup applications
A method, apparatus, and system for determining a consistency marker object representing the latest consistent recovery point for a disaster recovery (DR) process is disclosed. A request to move data from a local storage to a cloud storage for the DR process at a target point in time is received. Then, a replication completion status of a replication process indicating whether the replication is complete or not is determined. Next, the replication completion status indicating the replication process is complete is detected. Thereafter, in response to detecting the replication completion status, the consistency marker object having the latest consistent point is written in a preconfigured directory in a bucket to determine the consistent recovery point for a future DR process without having to perform lengthy validations for the data and meta.
1. A method of determining a consistency marker object representing a latest consistent recovery point for a disaster recovery (DR) process, the method comprising:
receiving a request to move data from a local storage to a cloud storage for the DR process at a target point in time;
determining a replication completion status of a replication process indicating whether the replication process is complete;
detecting the replication completion status indicating the replication process is complete;
in response to detecting the replication completion status, writing the consistency marker object having the latest consistent recovery point in a preconfigured directory in a cloud bucket for determining a consistent recovery point for a future DR process; and
determining whether a data movement process is complete and local objects including data and metadata are written to the cloud bucket.
2. The method of claim 1 , further comprising:
performing the DR process immediately from the latest consistent recovery point by selecting the consistency marker object as the consistent recovery point.
3. The method of claim 1 , further comprising: storing the local objects written in the cloud storage as cloud data objects in a translational map mapping the cloud data objects and corresponding local objects.
4. The method of claim 3 , wherein the determining of the replication completion status comprises:
performing a query for the replication completion status of each of the cloud data objects in the translational map;
pausing the query for the replication completion status for a predetermined amount of time if the replication completion status of one or more objects indicate that the replication process is not yet complete; and
removing each of the cloud data objects from the translational map if a replication status of each of the cloud data objects indicates the replication process is complete.
5. The method of claim 4 , wherein the query for the replication completion status is stopped when the replication status of all cloud data objects is complete or the replication status of one or more of the cloud data objects indicates that replication process associated with the cloud data objects has failed.
6. The method of claim 5 , wherein the query for the replication completion status is performed using an object reporting API of a cloud provider, the replication completion status being available via a GET object header or a HEAD request for the cloud data objects.
7. The method of claim 5 , wherein the query is performed using a lambda function in a cloud provider.
8. The method of claim 4 , further comprising:
performing, by a cloud provider, the replication process for the cloud data objects if the replication process has failed, wherein the performing of the replication process for the cloud data objects includes:
rewriting the cloud data objects associated with the failed replication process in a same path of the cloud storage to retrigger the replication process.
9. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations of determining a consistency marker object representing a latest consistent recovery point for a disaster recovery (DR) process, the operations comprising:
receiving a request to move data from a local storage to a cloud storage for the DR process at a target point in time;
determining a replication completion status of a replication process indicating whether the replication process is complete or not;
detecting the replication completion status indicating the replication process is complete;
in response to detecting the replication completion status, writing the consistency marker object having the latest consistent recovery point in a preconfigured directory in a cloud bucket to determine a consistent recovery point for a future DR process; and
determining whether a data movement process is complete and local objects including data and metadata are written to the cloud bucket.
10. The machine-readable medium of claim 9 , wherein the operations comprising further comprises:
performing the DR process immediately from the latest consistent recovery point by selecting the consistency marker object as the consistent recovery point.
11. The machine-readable medium of claim 9 , wherein the operations comprising further comprises: storing the local objects written in the cloud storage as cloud data objects in a translational map mapping the cloud data objects and corresponding local objects.
12. The machine-readable medium of claim 11 , wherein the determining of the replication completion status comprises:
performing a query for the replication completion status of each of the cloud data objects in the translational map;
pausing the query for the replication completion status for a predetermined amount of time if the replication completion status of one or more objects indicate that the replication process is not yet complete; and
removing each of the cloud data objects from the translational map if a replication status of each of the cloud data objects indicates the replication process is complete.
13. The machine-readable medium of claim 12 , wherein the query for the replication completion status is stopped when the replication status of all cloud data objects is complete or the replication status of one or more of the cloud data objects indicates that replication process associated with the cloud data objects has failed.
14. The machine-readable medium of claim 13 , wherein the query for the replication completion status is performed using an object reporting API of a cloud provider, the replication completion status being available via a GET object header or a HEAD request for the cloud data objects.
15. A data processing system, comprising:
a processor; and
a memory to store instructions, which when executed by the processor, cause the processor to perform operations of determining a consistency marker object representing a latest consistent recovery point for a disaster recovery (DR) process, the operations including:
receiving a request to move data from a local storage to a cloud storage for the DR process at a target point in time;
determining a replication completion status of a replication process indicating whether the replication process is complete or not;
detecting the replication completion status indicating the replication process is complete;
in response to detecting the replication completion status, writing the consistency marker object having the latest consistent recovery point in a preconfigured directory in a cloud bucket to determine a consistent recovery point for a future DR process; and
determining whether a data movement process is complete and local objects including data and metadata are written to the cloud bucket.
16. The data processing system of claim 15 , wherein the operations comprising further comprises:
performing the DR process immediately from the latest consistent recovery point by selecting the consistency marker object as the consistent recovery point.
17. The data processing system of claim 15 , wherein the operations comprising further comprises:
storing the local objects written in the cloud storage as cloud data objects in a translational map mapping the cloud data objects and corresponding local objects.
18. The data processing system of claim 17 , wherein the determining of the replication completion status comprises:
performing a query for the replication completion status of each of the cloud data objects in the translational map;
pausing the query for the replication completion status for a predetermined amount of time if the replication completion status of one or more objects indicate that the replication process is not yet complete; and
removing each of the cloud data objects from the translational map if a replication status of each of the cloud data objects indicates the replication process is complete.
19. The data processing system of claim 18 , wherein the query for the replication completion status is stopped when the replication status of all cloud data objects is complete or the replication status of one or more of the cloud data objects indicates that replication process associated with the cloud data objects has failed.
20. The data processing system of claim 19 , wherein the query for the replication completion status is performed using an object reporting API of a cloud provider, the replication completion status being available via a GET object header or a HEAD request for the cloud data objects.