Method or an apparatus to move perfect de-duplicated unique data from a source to destination storage tier
A data management device includes a persistent storage and a processor. The persistent storage includes an object storage that stores segments. The processor generates a collision free hash function based on the segments, generates a hash vector using the collision free hash function, deduplicates a portion of the segments associated with to-be-migrated files using the hash vector, and migrates the to-be-migrated files using the deduplicated portion of the segments to a remote storage.
1. A method for deduplicating data, comprising:
identifying in a persistent storage device, a first copy of a file segment;
obtaining, based on the identifying of the first copy of the file segment, a fingerprint associated with file segment;
identifying an indicator in a hash vector corresponding to the fingerprint, wherein the hash vector comprises a plurality of indicators;
determining that the indicator is set to a predetermined value, wherein the predetermined value indicates that the file segment is unique;
in response to determining that the indicator is set to the predetermined value:
storing the first copy of the file segment in a buffer;
modifying the indicator to a value other than the predetermined value;
after the modifying:
identifying, in the persistent storage device, a second copy of the file segment;
obtaining, based on the identifying of the second copy of the file segment, the fingerprint;
identifying the indicator in the hash vector corresponding to the fingerprint;
determining that the indicator is not set to the predetermined value;
in response to determining that the indicator is not set to the predetermined value, marking the second copy of the file segment as processed and not storing the second copy of the file segment in the buffer; and
initiating a deduplication procedure to replicate the first copy of the file segment from the persistent storage device to a remote storage device, wherein the deduplication procedure uses the buffer.
2. The method of claim 1 , wherein the persistent storage device comprises:
the first copy of the file segment; and
a second copy of the file segment, wherein the second copy of the file segment is not replicated to the remote storage device.
3. The method of claim 1 , further comprising:
prior to obtaining the fingerprint:
selecting the file segment; and
adding the indicator, to the hash vector, wherein adding the indicator comprises set the indicator the predetermined value.
4. The method of claim 3 , wherein selecting the file segment comprises:
identifying a file from a plurality of files;
making a determination that the file matches a criterion; and
in response to the determination that file matches the criterion, identifying, in the file, the file segment.
5. The method of claim 4 , wherein the criterion is one selected from the group consisting of file segment age, file segment storage date, and redundancy policy.
6. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for deduplicating data, the method comprising:
identifying in a persistent storage device, a first copy of a file segment;
obtaining, based on the identifying of the first copy of the file segment, a fingerprint associated with the file segment;
identifying an indicator in a hash vector corresponding to the fingerprint, wherein the hash vector comprises a plurality of indicators;
determining that the indicator is set to a predetermined value, wherein the predetermined value indicates that the file segment is unique;
in response to determining that the indicator is set to the predetermined value:
storing the first copy of the file segment in a buffer;
modifying the indicator to a value other than the predetermined value;
after the modifying:
identifying, in the persistent storage device, a second copy of the file segment;
obtaining, based on the identifying of the second copy of the file segment, the fingerprint;
identifying the indicator in the hash vector corresponding to the fingerprint;
determining that the indicator is not set to the predetermined value;
in response to determining that the indicator is not set to the predetermined value, marking the second copy of the file segment as processed and not storing the second copy of the file segment in the buffer; and
initiating a deduplication procedure to replicate the first copy of the file segment from the persistent storage device to a remote storage device, wherein the deduplication procedure uses the buffer.
7. The non-transitory computer readable medium of claim 6 , wherein the persistent storage device comprises:
the first copy of the file segment; and
a second copy of the file segment, wherein the second copy of the file segment is not replicated to the remote storage device.
8. The non-transitory computer readable medium of claim 6 , further comprising:
prior to obtaining the fingerprint:
selecting the file segment; and
adding the indicator, to the hash vector, wherein adding the indicator comprises set the indicator the predetermined value.
9. The non-transitory computer readable medium of claim 8 , wherein selecting the file segment comprises:
identifying a file from a plurality of files;
making a determination that the file matches a criterion; and
in response to the determination that file matches the criterion, identifying, in the file, the file segment.
10. The non-transitory computer readable medium of claim 9 , wherein the criterion is one selected from the group consisting of file segment age, file segment storage date, and redundancy policy.
11. A data management device, comprising:
persistent storage device; and
a processor programmed to:
identify, in a persistent storage device, a first copy of a file segment;
obtain, based on the identifying of the first copy of the file segment, a fingerprint associated with the file segment;
identify an indicator in a hash vector corresponding to the fingerprint, wherein the hash vector comprises a plurality of indicators;
determine that the indicator is set to a predetermined value, wherein the predetermined value indicates that the file segment is unique;
in response to determine that the indicator is set to the predetermined value:
store the first copy of the file segment in a buffer;
modify the indicator to a value other than the predetermined value;
after the modify:
identify, in the persistent storage device, a second copy of the file segment;
obtain, based on the identifying of the second copy of the file segment, the fingerprint;
identify the indicator in the hash vector corresponding to the fingerprint;
determine that the indicator is not set to the predetermined value;
in response to determining that the indicator is not set to the predetermined value, mark the second copy of the file segment as processed and not storing the second copy of the file segment in the buffer; and
initiate a deduplication procedure to replicate the first copy of the file segment from the persistent storage device to a remote storage device, wherein the deduplication procedure uses the buffer.
12. The data management device of claim 11 , wherein the persistent storage device comprises:
the first copy of the file segment; and
a second copy of the file segment, wherein the second copy of the file segment is not replicated to the remote storage device.
13. The data management device of claim 5 , wherein the processor is further configured to:
prior to obtaining the fingerprint:
select the file segment; and
add the indicator, to the hash vector, wherein adding the indicator comprises set the indicator the predetermined value.
14. The data management device of claim 13 , wherein selecting the file segment comprises:
identifying a file from a plurality of files;
making a determination that the file matches a criterion; and
in response to the determination that file matches the criterion, identifying, in the file, the file segment.
15. The data management device of claim 14 , wherein the criterion is one selected from the group consisting of file segment age, file segment storage date, and redundancy policy.