Method and system to verify integrity of a portion of replicated data
In general, embodiments of the invention relate to methods and systems for replicating data, which is stored in a source system, in a target system. More specifically, embodiments of the invention enable parallel transmission and verification of portions of the data. Once the portions of the data have been verified, embodiments of the invention combine the verified portions of the data to obtain the final combined data. The combined data is then verified.
1. A method for replicating data, the method comprising:
obtaining L0 fingerprints for a first portion of data;
storing the L0 fingerprints for the first portion of the data in a first set of L1 segments in persistent storage on a target system;
generating, by the target system, a first set of L1 fingerprints using the L1 segments;
storing the first set of L1 fingerprints in a first set of L2 segments in the persistent storage;
generating a first set of L2 fingerprints using the first set of L1 segments, wherein each L2 fingerprint in the first set of L2 fingerprints comprises a hash component and a checksum component;
storing the first set of L2 fingerprints in a first set of L3 segments in the persistent storage;
verifying, by the target system, the integrity of the first set of L2 segments using the first set of L2 fingerprints;
based on the verifying, merging the first portion of the data with a second portion of the data to obtain merged data on the target system, wherein the second portion of the data is verified independently from the first portion of the data; and
verifying the integrity of the merged data,
obtaining L0 fingerprints for the second portion of data;
storing the L0 fingerprints for the second portion for the data in a second set of L1 segments in the persistent storage;
generating, by the target system, a second set of L1 fingerprints using the second set of L1 segments;
storing the second set of L1 fingerprints in a second set of L2 segments in the persistent storage;
generating a second set of L2 fingerprints using the second set of L2 segments;
storing the second set of L2 fingerprints in a second set of L3 segments in the persistent storage;
wherein the verifying second portion of the data comprises verifying, by the target system, the integrity of the second set of L2 segments using the second set of L2 fingerprints, and
wherein obtaining the L0 fingerprints for the first portion of data is performed in parallel with obtaining the L0 fingerprints for the second portion of data,
wherein at least one file of the data to be replicated comprises the merged data and a third portion of data.
2. The method of claim 1 , further comprising:
determining, using the first set of L0 fingerprints, that a L0 segment corresponding to a L0 fingerprint in the first set of L0 fingerprints is not present on the target system;
in response to the determining, requesting the L0 segment from the source system;
in response to the sending, receiving the L0 segment from the source system and storing the L0 segment in the persistent storage.
3. The method of claim 1 , wherein the hash component comprises a hash value generated using a L2 segment in the first set of L2 segments and the checksum component comprises a checksum value generating using checksum values of all L1 segments referenced by the L2 segment.
4. The method of claim 1 , wherein verifying the integrity of the first portion of the data comprises obtaining at least a L2 checksum from a source system and comparing the obtained L2 checksum with at least one L2 checksum in the first set of L2 fingerprints.
5. The method of claim 1 , wherein verifying the integrity of the merged data comprises obtaining at least a root checksum from a source system and comparing the obtained root checksum with a root checksum of the merged data.
6. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for replicating data, the method comprising:
obtaining L0 fingerprints for a first portion of data;
storing the L0 fingerprints for the first portion of the data in a first set of L1 segments in persistent storage on a target system;
generating, by the target system, a first set of L1 fingerprints using the L1 segments;
storing the first set of L1 fingerprints in a first set of L2 segments in the persistent storage;
generating a first set of L2 fingerprints using the first set of L1 segments, wherein each L2 fingerprint in the first set of L2 fingerprints comprises a hash component and a checksum component;
storing the first set of L2 fingerprints in a first set of L3 segments in the persistent storage;
verifying, by the target system, the integrity of the first set of L2 segments using the first set of L2 fingerprints;
based on the verifying, merging the first portion of the data with a second portion of the data to obtain merged data on the target system, wherein the second portion of the data is verified independently from the first portion of the data; and
verifying the integrity of the merged data,
obtaining L0 fingerprints for the second portion of data;
storing the L0 fingerprints for the second portion for the data in a second set of L1 segments in the persistent storage;
generating, by the target system, a second set of L1 fingerprints using second set of L1 segments;
storing the second set of L1 fingerprints in a second set of L2 segments in the persistent storage;
generating a second set of L2 fingerprints using the second set of L2 segments;
storing the second set of L2 fingerprints in a second set of L3 segments in the persistent storage;
wherein the verifying second portion of the data comprises verifying, by the target system, the integrity of the second set of L2 segments using the second set of L2 fingerprints,
wherein obtaining the L0 fingerprints for the first portion of data is performed in parallel with obtaining the L0 fingerprints for the second portion of data,
wherein at least one file of the data to be replicated comprises the merged data and a third portion of data.
7. The non-transitory computer readable medium of claim 6 , the method further comprising:
determining, using the first set of L0 fingerprints, that a L0 segment corresponding to a L0 fingerprint in the first set of L0 fingerprints is not present on the target system;
in response to the determining, requesting the L0 segment from the source system;
in response to the sending, receiving the L0 segment from the source system and storing the L0 segment in the persistent storage.
8. The non-transitory computer readable medium of claim 6 , wherein the hash component comprises a hash value generated using a L2 segment in the first set of L2 segments and the checksum component comprises a checksum value generating using checksum values of all L1 segments referenced by the L2 segment.
9. The non-transitory computer readable medium of claim 6 , wherein verifying the integrity of the first portion of the data comprises obtaining at least a L2 checksum from a source system and comparing the obtained L2 checksum with at least one L2 fingerprint in the first set of L2 fingerprints.
10. The non-transitory computer readable medium of claim 6 , wherein verifying the integrity of the merged data comprises obtaining at least a root checksum from a source system and comparing the obtained root checksum with a root fingerprint of the merged data.
11. A system, comprising:
a source system comprising a processor and memory, wherein the memory comprises instructions, which when executed by the processor, perform a method, the method comprising data to:
sending L0 fingerprints for the first portion of data to a target system, and
sending at least one L2 checksum associated with the first portion of the data to the target system; and
the target system, operatively connected to the source system, comprising a second processor and a second memory, wherein the memory comprises instructions, which when executed by the second processor, perform a second method, the second method:
obtaining the L0 fingerprints for the first portion of data;
storing the L0 fingerprints for the first portion of the data in a first set of L1 segments in persistent storage on the target system;
generating a first set of L1 fingerprints using the L1 segments;
storing the first set of L1 fingerprints in a first set of L2 segments in the persistent storage;
generating a first set of L2 fingerprints using the first set of L1 segments, wherein each L2 fingerprint in the first set of L2 fingerprints comprises a hash component and a checksum component;
verifying the integrity of the first set of L2 segments using the at least one L2 checksum associated with the first portion of the data;
based on the verifying, merging the first portion of the data with the second portion of the data to obtain merged data, wherein the second portion of the data is verified independently from the first portion of the data; and
verifying the integrity of the merged data,
wherein method performed by the source system further comprises:
sending L0 fingerprints for the second portion of the data to the target system, and
sending at least one L2 checksum associated with the second portion of the data to the target system; and
wherein the second method performed by the target system further comprises:
obtaining L0 fingerprints for the second portion of data;
storing the L0 fingerprints for the second portion for the data in a second set of L1 segments in the persistent storage;
generating a second set of L1 fingerprints using the L1 segments;
storing the second set of L1 fingerprints in a second set of L2 segments in the persistent storage;
generating a second set of L2 fingerprints using the second set of L2 segments;
wherein the verifying second portion of the data comprises verifying the integrity of the second set of L2 segments using the at least one L2 checksum associated with the second portion of the data,
wherein the L0 fingerprints for the first portion of data are send in parallel with the L0 fingerprints for the second portion of data,
wherein at least one file of the data to be replicated comprises the merged data and a third portion of data.
12. The system of claim 11 , wherein the hash component comprises a hash value generated using a L2 segment in the first set of L2 segments and the checksum component comprises a checksum value generating using checksum values of all L1 segments referenced by the L2 segment.