Efficient segment detection for deduplication
View Patent ↗Mechanisms are provided for efficiently detecting segments for deduplication. Data is analyzed to determine file types and file components. File types such as images may have optimal data segment boundaries set at the file boundaries. Other file types such as container files are delayered to extract objects to set optimal data segment boundaries based on file type or based on the boundaries of the individual objects. Storage of unnecessary information is minimized in a deduplication dictionary while allowing for effective deduplication.
1. A method comprising:
determining that a first file received for deduplication at a deduplication system is a non-container file based on file type information associated with the first file;
identifying whether the first file would benefit from fine-grained segmentation based on file type information;
determining that a second file received for deduplication at the deduplication system is a container file based on file type information associated with the second file;
delayering the second file to extract a plurality of components;
setting segment boundaries for deduplication of the second file at component boundaries, wherein segment boundaries are used during deduplication to identify duplicate segments.
2. The method of claim 1 , wherein the second file is recursively parsed to determine whether the plurality of components include container files.
3. The method of claim 1 , wherein segment sizes are based on a dynamically tunable value if the first file would benefit from fine-grained segmentation.
4. The method of claim 1 , wherein segment sizes are based on a predetermined value based on file type if the first file would benefit from fine-grained segmentation.
5. The method of claim 1 , wherein the second file includes a plurality of container and non-container components.
6. The method of claim 1 , wherein the second file includes a plurality of non-container components that would benefit from fine-grained segmentation.
7. The method of claim 1 , wherein container files include a plurality of components.
8. The method of claim 7 , wherein a plurality of files including the first file and the second file are deduplicated by generating a plurality of filemaps.
9. The method of claim 7 , wherein a plurality of files including the first file and the second file are deduplicated by generating a plurality of datastore suitcases.
10. The method of claim 9 , wherein the plurality of datastore suitcases comprise a plurality of reference counts corresponding to a plurality of deduplicated data segments.
11. A deduplication system comprising:
an interface configured to receive a first file and a second file for deduplication at the deduplication system;
a processor configured to determine that the first file is a non-container file and the second file is a container file, wherein the processor determines that the first file would benefit from fine-grained segmentations based on file type information associated with the first file;
wherein the processor delayers the second file to extract a plurality of components and sets segment boundaries for deduplication of the second file at component boundaries, wherein segment boundaries are used during deduplication to identify duplicate segments.
12. The deduplication system of claim 11 , wherein the second file is recursively parsed to determine whether the plurality of components include container files.
13. The deduplication system of claim 11 , wherein segment sizes are based on a dynamically tunable value if the first file would benefit from fine-grained segmentation.
14. The deduplication system of claim 11 , wherein segment sizes are based on a predetermined value based on file type if the first file would benefit from fine-grained segmentation.
15. The deduplication system of claim 11 , wherein the second file includes a plurality of container and non-container components.
16. The deduplication system of claim 11 , wherein the second file includes a plurality of non-container components that would benefit from fine-grained segmentation.
17. The deduplication system of claim 11 , wherein container files include a plurality of components.
18. The deduplication system of claim 17 , wherein a plurality of files including the first file and the second file are deduplicated by generating a plurality of filemaps.
19. The deduplication system of claim 17 , wherein a plurality of files including the first file and the second file are deduplicated by generating a plurality of datastore suitcases.
20. A non-transitory computer readable medium comprising:
computer code for determining that a first file received for deduplication at a deduplication system is a non-container file based on file type information associated with the first file;
computer code for identifying whether the first file would benefit from fine-grained segmentation based on file type information;
computer code for determining that a second file received for deduplication at the deduplication system is a container file based on file type information associated with the second file;
computer code for delayering the second file to extract a plurality of components;
computer code for setting segment boundaries for deduplication of the second file at component boundaries, wherein segment boundaries are used during deduplication to identify duplicate segments.