Global deduplication
A first fingerprint corresponding to a first chunk associated with a stream of data is generated. It is determined that the first fingerprint matches a second fingerprint of a plurality of fingerprints listed in at least one entry in a deduplication map associated with a plurality of storage systems. A first storage system of the plurality of storage systems is located at a first geographic location and a second storage system of the plurality of storage systems is located at a second geographic location. The first chunk corresponding to the second fingerprint is stored by at least the second storage system. In response to a determination that the first fingerprint matches the second fingerprint, it is determined to store at the first storage system a local copy of the first chunk based in part on one or more deduplication factors. In response to the determination that the one or more deduplication factors indicate to store the local copy of the first chunk, the local copy of the first chunk is stored at the first storage system.
1. A system, comprising:
a processor configured to:
generate a first fingerprint corresponding to a first chunk associated with a stream of data;
determine that the first fingerprint matches a second fingerprint of a plurality of fingerprints listed in at least one entry in a deduplication map associated with a plurality of storage systems, wherein a first storage system of the plurality of storage systems is located at a first geographic location and a second storage system of the plurality of storage systems is located at a second geographic location, wherein the first chunk corresponding to the second fingerprint is stored by at least the second storage system;
in response to a determination that the first fingerprint matches the second fingerprint, determine to store at the first storage system a local copy of the first chunk based in part on one or more deduplication factors; and
in response to the determination that the one or more deduplication factors indicate to store the local copy of the first chunk, store at the first storage system the local copy of the first chunk; and
a memory coupled to the processor and configured to provide the processor with instructions.
2. The system of claim 1 , wherein the one or more deduplication factors include at least one of distance between the first storage system and one or more other storage systems included in the plurality of storage systems, availability of the plurality of storage systems, and/or a bandwidth of the plurality of storage systems.
3. The system of claim 1 , wherein the processor is further configured to determine whether the first fingerprint matches at least one fingerprint of the plurality of fingerprints listed in the at least one entry in the deduplication map associated with the plurality of storage systems.
4. The system of claim 3 , wherein the processor is further configured to add an entry in the deduplication map for the first fingerprint.
5. The system of claim 4 , wherein the entry includes a reference to the first fingerprint, an identifier of the first storage system, and/or a location within the first storage system.
6. The system of claim 5 , wherein the entry further includes an indication of whether the first storage system is enabled for inter-system deduplication.
7. The system of claim 1 , wherein the processor is further configured to propagate the deduplication map to one or more other storage systems of the plurality of storage systems.
8. The system of claim 7 , wherein the deduplication map is scheduled to be propagated as a background activity.
9. The system of claim 1 , wherein in response to the determination that the one or more deduplication factors indicate do not indicate to store the local copy of the first chunk, the processor is further configured to write a pointer to a second chunk corresponding to the second fingerprint.
10. The system of claim 9 , wherein the processor is further configured to update a reference counter associated with the second chunk corresponding to the second fingerprint.
11. The system of claim 1 , wherein the second storage system is associated with a cloud provider.
12. A method, comprising:
generating, by a first storage system, a first fingerprint corresponding to a first chunk associated with a stream of data;
determining, by the first storage system, that the first fingerprint matches a second fingerprint of a plurality of fingerprints listed in at least one entry in a deduplication map associated with a plurality of storage systems, wherein the first storage system of the plurality of storage systems is located at a first geographic location and a second storage system of the plurality of storage systems is located at a second geographic location, wherein the first chunk corresponding to the second fingerprint is stored by at least the second storage system;
in response to a determination that the first fingerprint matches the second fingerprint, determining, by the first storage system, to store a local copy of the first chunk based in part on one or more deduplication factors; and
in response to the determination that the one or more deduplication factors indicate to store the local copy of the first chunk, storing at the first storage system the local copy of the first chunk.
13. The method of claim 12 , wherein the one or more deduplication factors include at least one of distance between the first storage system and one or more other storage systems included in the plurality of storage systems, availability of the plurality of storage systems, and/or a bandwidth of the plurality of storage systems.
14. The method of claim 12 , further comprising determining whether the first fingerprint matches at least one fingerprint of the plurality of fingerprints listed in the at least one entry in the deduplication map associated with the plurality of storage systems.
15. The method of claim 12 , further comprising adding an entry in the deduplication map for the first fingerprint.
16. The method of claim 12 , further comprising propagating the deduplication map to one or more other storage systems of the plurality of storage systems.
17. The method of claim 16 , wherein the deduplication map is scheduled to be propagated as a background activity.
18. A computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
generating a first fingerprint corresponding to a first chunk associated with a stream of data;
determining that the first fingerprint matches a second fingerprint of a plurality of fingerprints listed in at least one entry in a deduplication map associated with a plurality of storage systems, wherein a first storage system of the plurality of storage systems is located at a first geographic location and a second storage system of the plurality of storage systems is located at a second geographic location, wherein the first chunk corresponding to the second fingerprint is stored by at least the second storage system;
in response to a determination that the first fingerprint matches the second fingerprint, determining to store at the first storage system a local copy of the first chunk based in part on one or more deduplication factors; and
in response to the determination that the one or more deduplication factors indicate to store the local copy of the first chunk, storing at the first storage system the local copy of the first chunk.
19. The computer program product of claim 18 , wherein the one or more deduplication factors include at least one of distance between the first storage system and one or more other storage systems included in the plurality of storage systems, availability of the plurality of storage systems, and/or a bandwidth of the plurality of storage systems.
20. The computer program product of claim 18 , further comprising computer instructions for determining whether the first fingerprint matches at least one fingerprint of the plurality of fingerprints listed in the at least one entry in the deduplication map associated with the plurality of storage systems.