Methods and apparatus for efficient compression and deduplication
View Patent ↗Mechanisms are provided for performing efficient compression and deduplication of data segments. Compression algorithms are learning algorithms that perform better when data segments are large. Deduplication algorithms, however, perform better when data segments are small, as more duplicate small segments are likely to exist. As an optimizer is processing and storing data segments, the optimizer applies the same compression context to compress multiple individual deduplicated data segments as though they are one segment. By compressing deduplicated data segments together within the same context, data reduction can be improved for both deduplication and compression. Mechanisms are applied to compensate for possible performance degradation.
1. A method, comprising:
partitioning a plurality of data files into a plurality of data segments for deduplication;
identifying a first subset of data segments from the plurality of data segments as infrequently accessed data segments;
identifying a second subset of data segments from the plurality of data segments as frequently accessed data segments;
compressing the first subset of data segments using a shared compression context, wherein a compressor examines patterns across the first subset of data segments to determine frequently occurring patterns during compression of the first subset of data segments; and
compressing the second subset of data segments using a plurality of non-shared compression contexts.
2. The method of claim 1 , wherein the plurality of data segments span a plurality of files.
3. The method of claim 1 , wherein the plurality of files are determined to be container or non-container files based on the file types associated with the files.
4. The method of claim 1 , wherein the plurality of data files are partitioned using segment sizes dependent on file type.
5. The method of claim 3 , wherein non-container files are evaluated to identify whether the non-container files would benefit from more fine-grained segmentation, wherein non-container files that would not benefit from more fine-grained segmentation have segment boundaries set to the file boundaries.
6. The method of claim 5 , wherein container files are recursively parsed to determine whether components of the container files are container or non-container components.
7. The method of claim 5 , wherein deduplicating the plurality of files comprises generating a plurality of filemaps corresponding to the plurality of files.
8. The method of claim 5 , wherein deduplicating the plurality of files comprises generating a plurality of datastore suitcases.
9. The method of claim 8 , wherein the datastore suitcase further comprises a plurality of reference counts corresponding to the plurality of deduplicated data segments.
10. The method of claim 9 , wherein the plurality of deduplicated data segments are determined to be infrequently accessed if they have low associated reference counts.
11. A system, comprising:
an interface configured to receive a plurality of data files;
a processor configured to partition a plurality of data files into a plurality of data segments for deduplication;
storage configured to maintain the plurality of data segments;
wherein a first subset of data segments from the plurality of data segments is identified as infrequently accessed data segments and a second subset of data segments from the plurality of data segments is identified as frequently accessed data segments, wherein the first subset of data segments is compressed using a shared compression context, wherein a compressor examines patterns across the first subset of data segments to determine frequently occurring patterns during compression of the first subset of data segments, and wherein the second subset of data segments is compressed using a plurality of non-shared compression contexts.
12. The system of claim 11 , wherein the plurality of data segments span a plurality of files.
13. The system of claim 11 , wherein the plurality of files are determined to be container or non-container files based on the file types associated with the files.
14. The system of claim 11 , wherein the plurality of data files are partitioned using segment sizes dependent on file type.
15. The system of claim 13 , wherein non-container files are evaluated to identify whether the non-container files would benefit from more fine-grained segmentation, wherein non-container files that would not benefit from more fine-grained segmentation have segment boundaries set to the file boundaries.
16. The system of claim 15 , wherein container files are recursively parsed to determine whether components of the container files are container or non-container components.
17. The system of claim 15 , wherein deduplicating the plurality of files comprises generating a plurality of filemaps corresponding to the plurality of files.
18. The system of claim 15 , wherein deduplicating the plurality of files comprises generating a plurality of datastore suitcases.
19. The system of claim 18 , wherein the datastore suitcase further comprises a plurality of reference counts corresponding to the plurality of deduplicated data segments.
20. A non-transitory computer readable medium, comprising:
computer code for partitioning a plurality of data files into a plurality of data segments for deduplication;
computer code for identifying a first subset of data segments from the plurality of data segments as infrequently accessed data segments;
computer code for identifying a second subset of data segments from the plurality of data segments as frequently accessed data segments;
computer code for compressing the first subset of data segments using a shared compression context, wherein a compressor examines patterns across the first subset of data segments to determine frequently occurring patterns during compression of the first subset of data segments; and
computer code for compressing the second subset of data segments using a plurality of non-shared compression contexts.