Efficient segment detection for deduplication
View Patent ↗Mechanisms are provided for efficiently detecting segments for deduplication. Data is analyzed to determine file types and file components. File types such as images may have optimal data segment boundaries set at the file boundaries. Other file types such as container files are delayered to extract objects to set optimal data segment boundaries based on file type or based on the boundaries of the individual objects. Storage of unnecessary information is minimized in a deduplication dictionary while allowing for effective deduplication.
1. A method, comprising:
determining file types associated with a plurality of files received for deduplication;
determining whether the plurality of files are container or non-container files based on the file types associated with the files;
evaluating non-container files to identify whether the non-container files would benefit from more fine-grained segmentation, wherein non-container files that would not benefit from more fine-grained segmentation have segment boundaries set to the file boundaries, wherein container files are delayered to extract a plurality of objects, wherein segment boundaries are set at object boundaries within the container files, wherein the segment boundaries are used to identify duplicates.
2. The method of claim 1 , further comprising recursively parsing container files to determine whether components of the container files are container or non-container components.
3. The method of claim 1 , wherein non-container files that would benefit from more fine-grained segmentation have segment sizes obtained based on file types, a predetermined value or a dynamically tunable value.
4. The method of claim 1 , wherein container files include a plurality of container and non-container components.
5. The method of claim 1 , wherein container files include a plurality of components that would benefit from more fine-grained segmentation.
6. The method of claim 1 , wherein container files include a plurality of components that would not benefit from more fine-grained segmentation.
7. The method of claim 1 , further comprising deduplicating the plurality of files.
8. The method of claim 7 , wherein deduplicating the plurality of files comprises generating a plurality of filemaps corresponding to the plurality of files.
9. The method of claim 7 , wherein deduplicating the plurality of files comprises generating a plurality of datastore suitcases.
10. The method of claim 9 , wherein the datastore suitcase further comprises a plurality of reference counts corresponding to a plurality of deduplicated data segments.
11. A system, comprising:
memory configured to hold a mapping associating file types with a plurality of segment sizes;
an interface configured to receive a plurality of files for deduplication;
a processor configured to determine whether the plurality of files are container or non-container files based on the file types associated with the file and determine whether the non-container files would benefit from more fine-grained segmentation, wherein non-container files that would not benefit from more fine-grained segmentation have segment boundaries set to the file boundaries, wherein container files are delayered to extract a plurality of objects, wherein segment boundaries are set at object boundaries within the container files, wherein the segment boundaries are used to identify duplicates.
12. The system of claim 11 , wherein container files are recursively parsed to determine whether components of the container files are container or non-container components.
13. The system of claim 11 , wherein non-container files that would benefit from more fine-grained segmentation have segment sizes obtained based on file types, a predetermined value or a dynamically tunable value.
14. The system of claim 11 , wherein container files include a plurality of container and non-container components.
15. The system of claim 11 , wherein container files include a plurality of components that would benefit from more fine-grained segmentation.
16. The system of claim 11 , wherein container files include a plurality of components that would not benefit from more fine-grained segmentation.
17. The system of claim 11 , further comprising deduplicating the plurality of files.
18. The system of claim 17 , wherein deduplicating the plurality of files comprises generating a plurality of filemaps corresponding to the plurality of files.
19. The system of claim 17 , wherein deduplicating the plurality of files comprises generating a plurality of datastore suitcases.
20. A computer readable medium having computer code embodied therein, the computer readable medium comprising:
computer code for determining file types associated with a plurality of files received for deduplication;
computer code for determining whether the plurality of files are container or non-container files based on the file types associated with the files;
computer code for evaluating non-container files to identify whether the non-container files would benefit from more fine-grained segmentation, wherein non-container files that would not benefit from more fine-grained segmentation have segment boundaries set to the file boundaries, wherein container files are delayered to extract a plurality of objects, wherein segment boundaries are set at object boundaries within the container files, wherein the segment boundaries are used to identify duplicates.