Efficiently estimating data compression ratio of ad-hoc set of files in protection storage filesystem with stream segmentation and data deduplication
In one example, a method for processing data includes receiving information that identifies an ad hoc group of size ‘n’ of files F 1 . . . F n , each file F including a respective file sequence S that includes K data segments. Next, each file sequence S is sampled to obtain a sequence SS of data segments from the file sequence S, and a non-random sampling of data segments is sampled from each sequence SS to obtain a set SSU of the sequence SS. The data segments of each set SSU are then sampled to obtain a sample subset SSUS of the set SSU, and a compression ratio is determined for each data segment in each sample subset SSUS. Finally, an average data compression R F1 . . . Fn is estimated and output for the files F in the group of size ‘n’, based on the compression ratios.
1. A method, comprising:
receiving a saveset at a storage node, the saveset comprising data that has been segmented and deduplicated;
improving data storage efficiency by compressing the segmented and deduplicated data, when the segmented and deduplicated data has not already been compressed;
writing the compressed data to storage;
receiving, from a user, a selection that identifies an ad hoc group of size ‘n’ of files F 1 . . . F n that are included in the stored data, each file F including a respective file sequence S that includes K data segments;
sampling each file sequence S to obtain a sequence SS of data segments from the file sequence S;
performing a non-random sampling of data segments from each sequence SS to obtain a set SSU of the sequence SS;
sampling the data segments of each set SSU to obtain a sample subset SSUS of the set SSU, wherein sampling the data segments of each set SSU and sampling the file sequences S reduces a number of Input/Output (IO) operations to disk/persistent storage that would otherwise be required by the method;
determining a compression ratio for each data segment in each sample subset SSUS;
estimating and outputting an average data compression R F1 . . . F n for the files F in the group of size ‘n’, based on the compression ratios; and
using the average data compression R F1 . . . F n to determine how much storage space is occupied by the files F.
2. The method as recited in claim 1 , wherein the non-random sampling of data segments from each sequence SS is based on the content of the sampled data segments.
3. The method as recited in claim 1 , wherein estimation of average data compression R F1 . . . F n for the files F in the group of size ‘n’ is performed at any time T.
4. The method as recited in claim 1 , wherein one or more of the files F includes deduplicated data.
5. The method as recited in claim 1 , further comprising using the average data compression R F1 . . . F n information as a basis to take one or more actions to be taken regarding data storage space in a datacenter, and wherein the actions to be taken regarding data storage space in the datacenter comprise changing a storage space efficiency of the datacenter.
6. The method as recited in claim 1 , wherein no data compression measurement has previously been performed respecting the sampled data segments S.
7. The method as recited in claim 1 , further comprising:
when retrieving compression information for one data segment from disk, the one data segment being included in a compression block, loading compression information for one or more other data segments within the same compression block into a lock-free, low memory overhead cache.
8. The method as recited in claim 1 , wherein a number of data segments sampled from each of the segment sets SSU is determined by applying a sampling ratio 1/M to each segment set SSU.
9. The method as recited in claim 1 , wherein performing a non-random sampling of data segments from each sequence SS to obtain a set SSU of sequence SS comprises identifying a set of unique data segments in the sequence SS.
10. A method for processing data, comprising:
deduplicating data segments of a saveset;
compressing data segments of the deduplicated saveset;
writing, or causing the writing of, the deduplicated and compressed saveset data segments to storage; and
performing, or causing the performance of, the following:
defining ad hoc a group of size ‘n’ of files F 1 . . . F n , one or more of which is included in the saveset; and
estimating an average data compression R F1 . . . F n for the files F in the group of size ‘n’, wherein some data of one or more of the files F in the saveset is undeduplicated data.
11. The method as recited in claim 10 , further comprising:
for each file F, identifying a sequence SS of data segments from a corresponding file sequence S 1 . . . S K ;
performing a non-random sampling of data segments from each sequence SS to obtain a set SSU of the sequence SS;
sampling the data segments of each set SSU to obtain a sample subset SSUS of the set SSU;
determining a compression ratio for each data segment in each sample subset SSUS; and
estimating and outputting an average data compression R F1 . . . F n for the files F in the group of size ‘n’, based on the compression ratios.
12. The method as recited in claim 11 , wherein a number of data segments sampled from each of the segment sets SSU is determined by applying a sampling ratio 1/M to each segment set SSU.
13. The method as recited in claim 10 , further comprising reporting the average data compression R F1 . . . F n .
14. The method as recited in claim 10 , further comprising taking any one or more of the following actions, using the average data compression R F1 . . . F n information:
changing a storage space efficiency of a datacenter;
identifying one or more files for movement to a different storage medium that is relatively less expensive than a storage medium in which the one or more files are then stored; or
identifying one or more files whose compression may be improved by an alternative to the compression method that was previously employed on the one or more files.
15. A non-transitory storage medium having stored therein computer-executable instructions which, when executed by one or more hardware processors, processes data by performing the following processes:
receiving a saveset at a storage node, the saveset comprising data that has been segmented and deduplicated;
improving data storage efficiency by compressing the segmented and deduplicated data, when the segmented and deduplicated data has not already been compressed;
writing the compressed data to storage;
receiving, from a user, a selection that identifies an ad hoc group of size ‘n’ of files F 1 . . . F n that are included in the stored data, each file F including a respective file sequence S that includes K data segments;
sampling each file sequence S to obtain a sequence SS of data segments from the file sequence S;
performing a non-random sampling of data segments from each sequence SS to obtain a set SSU of the sequence SS;
sampling the data segments of each set SSU to obtain a sample subset SSUS of the set SSU, wherein sampling the data segments of each set SSU and sampling the file sequences S reduces a number of Input/Output (IO) operations to disk/persistent storage that would otherwise be required by performance of the processes;
determining a compression ratio for each data segment in each sample subset SSUS;
estimating and outputting an average data compression R F1 . . . F n for the files F in the group of size ‘n’, based on the compression ratios; and
using the average data compression R F1 . . . F n to determine how much storage space is occupied by the files F.
16. The non-transitory storage medium of claim 15 , wherein the non-random sampling of data segments from each sequence SS is based on the content of the sampled data segments.
17. The non-transitory storage medium of claim 15 , wherein estimation of average data compression R F1 . . . F n for the files F in the group of size ‘n’ is performed at any time T.
18. The non-transitory storage medium of claim 15 , wherein one or more of the files F includes deduplicated data.