Duplication between disparate deduplication systems
Disclosed herein are methods, systems, and processes to improve the duplication of data between disparate deduplication systems. Source fingerprints are generated for data blocks using a source fingerprint algorithm at a source deduplication system. The source fingerprints and previously-generated source fingerprints are used to determine whether the data blocks are new or modified. If the data blocks are new or modified, target fingerprints are generated for the data blocks using a target fingerprint algorithm associated with a target deduplication system. The target fingerprints are sent to the target deduplication system.
1. A method comprising:
in response to a determination that a data block is new or has been modified,
storing, at a source deduplication system, a source fingerprint in an entry of a fingerprint translation table, wherein
the source fingerprint was generated for the data block using a source fingerprint algorithm, and
storing, at the source deduplication system, a target fingerprint in the entry of the fingerprint translation table, wherein
the target fingerprint was generated for the data block using a target fingerprint algorithm; and
sending the fingerprint translation table to a target deduplication system, wherein the target fingerprint algorithm is associated with the target deduplication system.
2. The method of claim 1 , further comprising:
accessing the target fingerprint algorithm, wherein
the accessing is performed by the source deduplication system.
3. The method of claim 2 , wherein the accessing comprises:
determining whether the target fingerprint algorithm is available at the source deduplication system; and
in response to a determination that the target fingerprint algorithm is not available at the source deduplication system,
sending a request for the target fingerprint algorithm to the target deduplication system, and
receiving the target fingerprint algorithm from the target fingerprint algorithm.
4. The method of claim 2 , wherein the accessing comprises:
determining whether the data block is new or has been modified; and
in response to a determination that the data block is not new and has not been modified,
determining whether the source fingerprint matches an existing source fingerprint in the fingerprint translation table.
5. The method of claim 1 , further comprising:
determining whether the source fingerprint matches any existing source fingerprints in the fingerprint translation table; and
in response to a determination that the source fingerprint matches an existing source fingerprint in the fingerprint translation table,
incrementing a reference count in an entry of the fingerprint translation table corresponding to the existing source fingerprint.
6. The method of claim 5 , wherein the incrementing produces an incremented reference count, and the method further comprises:
sending the incremented reference count to the target deduplication system.
7. The method of claim 1 , further comprising:
generating the source fingerprint using a source fingerprint algorithm; and
generating the target fingerprint using a target fingerprint algorithm, wherein
the generating the source fingerprint and the generating the target fingerprint are performed by the source deduplication system.
8. The method of claim 1 , further comprising:
receiving the fingerprint translation table at the target deduplication system;
determining whether to request one or more data blocks from the source deduplication system, wherein
the determining uses the fingerprint translation table; and
in response to a determination that the one or more data blocks should be requested from the source deduplication system, requesting the one or more data blocks from the source deduplication system.
9. The method of claim 8 , wherein the determining whether to request the one or more data blocks from the source deduplication system further comprises:
determining whether any target fingerprints in the fingerprint translation table are not in a target fingerprint database, wherein
the target fingerprint database is maintained at the target deduplication system; and
in response to a determination that one or more target fingerprints in the fingerprint translation table are not in the target fingerprint database, performing the requesting.
10. The method of claim 9 , further comprising:
in response to a determination that the one or more target fingerprints in the fingerprint translation table are in the target fingerprint database, requesting metadata associated with the one or more data blocks.
11. The method of claim 9 , further comprising:
updating a target catalog to include information regarding the one or more data blocks.
12. The method of claim 1 , wherein each entry of the fingerprint translation table comprises a duplicate field that indicates whether a data block is new.
13. The method of claim 1 , wherein
the target fingerprint is received from the target deduplication system.
14. A computer system, implementing a source deduplication system, comprising:
one or more processors;
a network interface, coupled to the one or more processors and configured to facilitate communications with a network;
a computer-readable storage medium coupled to the one or more processors; and
a plurality of instructions, encoded in the computer-readable storage medium and configured to cause the one or more processors to
in response to a determination that a data block is new or has been modified,
store, at the source deduplication system, a source fingerprint in an entry of a fingerprint translation table, wherein
the source fingerprint was generated for the data block using a source fingerprint algorithm, and
store, at the source deduplication system, a target fingerprint in the entry of the fingerprint translation table, wherein
the target fingerprint was generated for the data block using a target fingerprint algorithm, and
send the fingerprint translation table to a target deduplication system, via the network, wherein
the target fingerprint algorithm is associated with the target deduplication system.
15. The computer system of claim 14 , wherein the plurality of instructions is further configured to cause the one or more processors to:
access the target fingerprint algorithm, wherein
the plurality of instructions further configured to cause the one or more processors to access the target fingerprint algorithm comprises one or more instructions configured to cause the one or more processors to
determine whether the target fingerprint algorithm is available at the source deduplication system, and
in response to a determination that the target fingerprint algorithm is not available at the source deduplication system,
send a request for the target fingerprint algorithm to the target deduplication system, and
receive the target fingerprint algorithm from the target fingerprint algorithm.
16. The computer system of claim 14 , wherein the plurality of instructions is further configured to cause the one or more processors to:
determine whether the source fingerprint matches any existing source fingerprints in the fingerprint translation table; and
in response to a determination that the source fingerprint matches an existing source fingerprint in the fingerprint translation table,
producing an incremented reference count by incrementing a reference count in an entry of the fingerprint translation table corresponding to the existing source fingerprint, and
send the incremented reference count to the target deduplication system.
17. The computer system of claim 14 , wherein
each entry of the fingerprint translation table comprises a duplicate field that indicates whether a data block is new.
18. A computer program product comprising:
a plurality of instructions, comprising
a first set of instructions, executable by a processor of a computer system implementing a source deduplication system, configured to, in response to a determination that a data block is new or has been modified,
store, at the source deduplication system, a source fingerprint in an entry of a fingerprint translation table, wherein
the source fingerprint was generated for the data block using a source fingerprint algorithm, and
store, at the source deduplication system, a target fingerprint in the entry of the fingerprint translation table, wherein
the target fingerprint was generated for the data block using a target fingerprint algorithm, and
a second set of instructions, executable on the computer system, configured to send the fingerprint translation table to a target deduplication system, wherein
the target fingerprint algorithm is associated with the target deduplication system; and
a computer-readable storage medium, wherein the plurality of instructions are encoded in the computer-readable storage medium.
19. The computer program product of claim 18 , wherein the instructions further comprise:
a third set of instructions, executable by the processor, configured to access the target fingerprint algorithm, wherein
the third set of instructions comprise
a first subset of instructions, executable by the processor, configured to determine whether the target fingerprint algorithm is available at the source deduplication system, and
a second subset of instructions, executable by the processor, configured to, in response to a determination that the target fingerprint algorithm is not available at the source deduplication system,
send a request for the target fingerprint algorithm to the target deduplication system, and
receive the target fingerprint algorithm from the target fingerprint algorithm.
20. The computer program product of claim 18 , wherein the instructions further comprise:
a third set of instructions, executable by the processor, configured to determine whether the source fingerprint matches any existing source fingerprints in the fingerprint translation table; and
a fourth set of instructions, executable by the processor, configured to, in response to a determination that the source fingerprint matches an existing source fingerprint in the fingerprint translation table,
produce an incremented reference count by incrementing a reference count in an entry of the fingerprint translation table corresponding to the existing source fingerprint, and
send the incremented reference count to the target deduplication system.