Methods and systems for data resynchronization in a replication environment
Methods, computer program products, computer systems, and the like are disclosed that provide for scalable deduplication in an efficient and effective manner. For example, such methods, computer program products, and computer systems can include determining whether a source data store and a replicated data store are unsynchronized and, in response to a determination that the source data store and the replicated data store are unsynchronized, performing a resynchronization operation. The source data stored in the source data store is replicated to replicated data in the replicated data store. The resynchronization operation resynchronizes the source data and the replicated data.
1. A method comprising:
generating an update set, wherein
the update set is generated by a source data mover at a source site,
the update set comprises one or more replication updates, and
the update set comprises data of the one or more replication updates and associated metadata of the one or more replication updates;
generating a restore identifier at the source site, wherein
the restore identifier facilitates resynchronization of data stored at the source site with data stored at a target site, in event of a restoration operation, by facilitating identification of that portion of the data stored at the target site that has changed since a beginning of a data synchronization interval associated with the restore identifier, using a data structure for synchronization;
sending the update set and the restore identifier to a target data mover at the target site, wherein
the sending is performed by the source data mover; and
storing the restore identifier.
2. The method of claim 1 , wherein the storing comprises:
storing the update set at the target site; and
persisting the restore identifier at the target site, wherein
the restore identifier is stored at the source site as a current restore identifier.
3. The method of claim 2 , wherein
the storing the restore identifier is performed in response to receipt, by the source data mover, of an indication that the update set was successfully received by the target data mover.
4. The method of claim 2 , wherein the storing comprises:
writing the restore identifier to a dirty region log at the source site, wherein
the writing overwrites a previous restore identifier with the restore identifier, and
the previous restore identifier was the current restore identifier prior to the writing.
5. The method of claim 1 , further comprising:
storing the update set at the target site;
persisting the restore identifier at the target site, wherein
the restore identifier is stored at the source site;
comparing the restore identifier and a current restore identifier, wherein
the restore identifier and the current restore identifier are ones of a plurality of restore identifiers, and
the current restore identifier was stored prior to the generating the restore identifier; and
in response to the comparing indicating that the current restore identifier is subsequent to the restore identifier, causing the source data mover to indicate performance of the restore operation at the source site, to the target data mover.
6. The method of claim 5 , wherein
further in response to the comparing indicating that the restore identifier is equal or subsequent to the current restore identifier, preventing performance of the sending and the persisting.
7. The method of claim 5 , wherein
in response to the comparing indicating that the restore identifier is subsequent to the current restore identifier, performing the sending and the persisting.
8. The method of claim 5 , wherein
the restore operation is at least one of
a restore operation from a backup operation performed at the source site, or
a restore operation from a snapshot operation performed at the source site.
9. The method of claim 5 , further comprising:
during the restoration operation, preventing performance of at least one or a migration operation or a takeover operation.
10. The method of claim 1 , wherein
the restore identifier is one of a plurality of restore identifiers that are generated, and
a new restore identifier of the plurality of restore identifiers is generated with a periodicity that is based, at least in part, on at least one of
an amount of data replicated,
an active replication load, and/or
a time interval.
11. The method of claim 1 , wherein
the associated metadata comprises the restore identifier, and
the restore identifier is sent as part of the update set.
12. The method of claim 1 , wherein
the associated metadata further comprises:
logical block number information,
size information, and
physical block number information.
13. The method of claim 1 , wherein
the restore identifier is at least one of
a version,
a counter, or
a universally-unique identifier.
14. The method of claim 1 , further comprising:
maintaining an active bitmap and a synchronization bitmap, wherein
source data is maintained at the source site,
replicated data is maintained at the target site, and
the active bitmap and the synchronization bitmap are maintained such that at least one of the following is performed
in a running state, the active bitmap is updated and the synchronization bitmap is empty,
in a disconnected state, the active bitmap is updated and the synchronization bitmap is empty, or
during a re-connection operation,
the active bitmap is emptied by merging the active bitmap into the synchronization bitmap, and
the synchronization bitmap is cleared by resynchronizing the source data and the replicated data.
15. A method comprising:
receiving an update set and a restore identifier from a source data mover, wherein
the update set and the restore identifier are received by a target data mover at a target site,
the update set was generated from source data at a source site by the source data mover,
a replicated copy of the source data is maintained at the target site as replicated data,
the update set comprises one or more replication updates, and
the update set comprises data of the one or more replication updates and associated metadata of the one or more replication updates;
comparing the restore identifier to a persisted restore identifier;
in response to the comparing indicating that the restore identifier is subsequent to the persisted restore identifier,
persisting the restore identifier at the target site, and
applying the update set to the replicated data; and
in response to the comparing indicating that the persisted restore identifier is subsequent to the restore identifier, awaiting receipt of a synchronization bitmap.
16. The method of claim 15 , further comprising:
receiving the synchronization bitmap, wherein
the synchronization bitmap comprises a plurality of bits;
receiving a plurality of checksums, wherein
each checksum of the plurality of checksums is a checksum of a unit of source data represented by a bit of the plurality of bits; and
for each bit of the plurality of bits,
comparing a corresponding checksum of the plurality of checksums corresponding to the each bit and a checksum of a unit of target data corresponding to the unit of source data represented by the each bit, and
in response to the corresponding checksum and the checksum of the unit of target data corresponding to the unit of source data represented by the each bit not matching, sending the unit of target data to the source data mover.
17. The method of claim 15 , further comprising:
in response to the corresponding checksum and the checksum matching, skipping the unit of target data.
18. The method of claim 15 , wherein
the associated metadata further comprises:
logical block number information,
size information, and
physical block number information.
19. The method of claim 15 , wherein
the restore identifier is at least one of
a version,
a counter, or
a universally-unique identifier.
20. The method of claim 15 , wherein
the associated metadata comprises the restore identifier, and
restore identifier is sent as part of the update set.
21. A non-transitory computer-readable storage medium, comprising program instructions, which, when executed by one or more processors of a computing system, perform a method comprising:
generating an update set, wherein
the update set is generated by a source data mover at a source site,
the update set comprises one or more replication updates, and
the update set comprises data of the one or more replication updates and associated metadata of the one or more replication updates;
generating a restore identifier at the source site, wherein
the restore identifier facilitates resynchronization of data stored at the source site with data stored at a target site, in event of a restoration operation, by facilitating identification of that portion of the data stored at the target site that has changed since a beginning of a data synchronization interval associated with the restore identifier, using a data structure for synchronization;
sending the update set and the restore identifier to a target data mover at the target site, wherein
the sending is performed by the source data mover; and
storing the restore identifier.