Granted Patent B2

US 11,662,907 · App. 16/888,818 · Granted May 30, 2023

Inventors: Chao Lin (Chengdu, CN); Yuting Zhang (Chengdu, CN); Qianyun Cheng (Chengdu, CN)

Assignee: EMC IP HOLDING COMPANY LLC

G06F3/0608G06F3/067G06F3/0641G06F3/0647

Abstract

Claims (75)

1. A storage management method, comprising:

determining at least one count corresponding to at least one data segment of a file in a file set, the file set being stored in a local storage device, the at least one count indicating a number of occurrences of the at least one data segment in the file set, wherein each file in the file set has a Merkel tree structure which is represented by a fingerprint index comprising of hash fingerprints of the at least one data segment and the metadata for the at least one data segment;

determining, using the fingerprint index, a deduplication ratio of the file based on the at least one count;

migrating the file from the local storage device to a remote storage device according to a determination that the deduplication ratio of the file is lower than a threshold,

wherein the threshold is determined based on a total deduplication ratio of the file set,

wherein the total deduplication ratio reflects an average level of deduplication ratios of all files in the file set,

wherein migrating the file to the remote storage device comprises:

copying the at least one data segment of the file from the local storage device to the remote storage device;

copying the metadata of the file from the local storage device to the remote storage device, the metadata indicating that the file includes the at least one data segment; and

deleting the at least one data segment from the local storage device without deleting the metadata,

determining, after migrating the file from the local storage device to the remote storage device, that an incremental backup file for the file is to be stored;

storing, in the remote storage device, different data segments of the incremental backup file relative to the file; and

storing the metadata of the incremental backup file in the local storage device, the metadata indicating data segments included in the incremental backup file.

2. The method of claim 1 , wherein determining at least one count comprises:

determining a set of data segments included in the file set;

determining a count set corresponding to the set of data segments, each count in the count set indicating a number of occurrences of one data segment in the set of data segments in the file set; and

determining, from the count set, the at least one count corresponding to the at least one data segment of the file.

3. The method of claim 2 , further comprising:

performing a garbage collection process of the local storage device based on the determined set of data segments and the determined count set.

4. The method of claim 1 ,

wherein the at least one data segment comprises a plurality of data segments, and

wherein determining the deduplication ratio comprises:

determining multiple numbers of occurrences of multiple different data segments of the plurality of data segments in the file;

based on the multiple numbers of occurrences and multiple counts corresponding to the plurality of data segments, determining multiple duplication ratios corresponding to the multiple different data segments; and

determining the deduplication ratio based on the number of the plurality of data segments and the multiple duplication ratios.

5. The method of claim 1 ,

wherein the total deduplication ratio of the file set is determined based on a logical size and a physical size of the file set.

6. The method of claim 1 , wherein the method is performed during a garbage collection process of the local storage device.

7. The method of claim 1 , wherein the remote storage device comprises a cloud storage device.

8. An electronic device, comprising:

at least one processor; and

at least one memory storing computer program instructions, the at least one memory and the computer program instructions being configured, together with the at least one processor, to cause the electronic device to perform a process, the process comprising:

determining at least one count corresponding to at least one data segment of a file in a file set, the file set being stored in a local storage device, the at least one count indicating a number of occurrences of the at least one data segment in the file set, wherein each file in the file set has a Merkel tree structure which is represented by a fingerprint index comprising of hash fingerprints of the at least one data segment and the metadata for the at least one data segment;

determining, using the fingerprint index, a deduplication ratio of the file based on the at least one count;

migrating the file from the local storage device to a remote storage device according to a determination that the deduplication ratio of the file is lower than a threshold,

wherein the threshold is determined based on a total deduplication ratio of the file set,

wherein the total deduplication ratio reflects an average level of deduplication ratios of all files in the file set,

wherein migrating the file to the remote storage device comprises:

copying the at least one data segment of the file from the local storage device to the remote storage device;

copying the metadata of the file from the local storage device to the remote storage device, the metadata indicating that the file includes the at least one data segment; and

deleting the at least one data segment from the local storage device without deleting the metadata,

determining, after migrating the file from the local storage device to the remote storage device, that an incremental backup file for the file is to be stored;

storing, in the remote storage device, different data segments of the incremental backup file relative to the file; and

storing the metadata of the incremental backup file in the local storage device, the metadata indicating data segments included in the incremental backup file.

9. The electronic device of claim 8 , wherein the electronic device is caused to determine the at least one count by:

determining a set of data segments included in the file set;

determining a count set corresponding to the set of data segments, each count in the count set indicating a number of occurrences of one data segment in the set of data segments in the file set; and

determining, from the count set, the at least one count corresponding to the at least one data segment of the file.

10. The electronic device of claim 9 , wherein the at least one memory and the computer program instructions are further configured to, together with the at least one processor, cause the electronic device to:

performing a garbage collection process of the local storage device based on the determined set of data segments and the determined count set.

11. The electronic device of claim 8 , wherein the at least one data segment includes a plurality of data segments, and wherein the electronic device is caused to determine the deduplication ratio by:

determining multiple numbers of occurrences of multiple different data segments of the plurality of data segments in the file;

based on the multiple numbers of occurrences and multiple counts corresponding to the plurality of data segments, determining multiple duplication ratios corresponding to the multiple different data segments; and

determining the deduplication ratio based on the number of the plurality of data segments and the multiple duplication ratios.

12. The electronic device of claim 8 , wherein

the total deduplication ratio of the file set is determined based on a logical size and a physical size of the file set.

13. The electronic device of claim 8 , wherein the electronic device is caused to perform the process during a garbage collection process of the local storage device.

14. The electronic device of claim 8 , wherein the remote storage device comprises a cloud storage device.

15. A computer program product that is tangibly stored in a non-volatile computer-readable medium and comprises machine-executable instructions that, when executed, cause a machine to execute the steps of a method, the method comprising:

determining at least one count corresponding to at least one data segment of a file in a file set, the file set being stored in a local storage device, the at least one count indicating a number of occurrences of the at least one data segment in the file set wherein each file in the file set has a Merkel tree structure which is represented by a fingerprint index comprising of hash fingerprints of the at least one data segment and the metadata for the at least one data segment;

determining, using the fingerprint index, a deduplication ratio of the file based on the at least one count, the deduplication ratio indicating an overlapping level of the file with other files in the file set;

migrating the file from the local storage device to a remote storage device according to a determination that the deduplication ratio of the file is lower than a threshold,

wherein the threshold is determined based on a total deduplication ratio of the file set,

wherein the total deduplication ratio reflects an average level of deduplication ratios of all files in the file set,

wherein migrating the file to the remote storage device comprises:

copying the at least one data segment of the file from the local storage device to the remote storage device;

copying the metadata of the file from the local storage device to the remote storage device, the metadata indicating that the file includes the at least one data segment; and

deleting the at least one data segment from the local storage device without deleting the metadata,

determining, after migrating the file from the local storage device to the remote storage device, that an incremental backup file for the file is to be stored;

storing, in the remote storage device, different data segments of the incremental backup file relative to the file; and

storing the metadata of the incremental backup file in the local storage device, the metadata indicating data segments included in the incremental backup file.

16. The computer program product of claim 15 , wherein determining at least one count comprises:

determining a set of data segments included in the file set;

determining a count set corresponding to the set of data segments, each count in the count set indicating a number of occurrences of one data segment in the set of data segments in the file set; and

determining, from the count set, the at least one count corresponding to the at least one data segment of the file.

Assignments (9)

Priority Claims (1)

Continuity (1)