Methods and apparatus for efficient compression and deduplication
View Patent ↗Mechanisms are provided for performing efficient compression and deduplication of data segments. Compression algorithms are learning algorithms that perform better when data segments are large. Deduplication algorithms, however, perform better when data segments are small, as more duplicate small segments are likely to exist. As an optimizer is processing and storing data segments, the optimizer applies the same compression context to compress multiple individual deduplicated data segments as though they are one segment. By compressing deduplicated data segments together within the same context, data reduction can be improved for both deduplication and compression. Mechanisms are applied to compensate for possible performance degradation.
1. A method, comprising:
receiving a plurality of data files;
partitioning the plurality of data files into a plurality of data segments for deduplication, wherein a segment size is determined using file type information associated with the plurality of data files;
identifying a plurality of deduplicated data segments configured for storage at a storage system;
compressing a plurality of deduplicated data segments using a common compression context, wherein the plurality of deduplicated data segments are compressed upon determining that the plurality of deduplicated data segments are infrequently accessed, wherein a compressor examines patterns across the plurality of infrequently accessed deduplicated data segments to determine frequently occurring patterns, wherein a single deduplicated data segment is compressed using a non-shared compression context upon determining that the single deduplicated data segment is accessed frequently.
2. The method of claim 1 , wherein the plurality of data segments span a plurality of files.
3. The method of claim 1 , wherein the plurality of files are determined to be container or non-container files based on the file types associated with the files.
4. The method of claim 3 , wherein non-container files are evaluated to identify whether the non-container files would benefit from more fine-grained segmentation, wherein non-container files that would not benefit from more fine-grained segmentation have segment boundaries set to the file boundaries.
5. The method of claim 4 , wherein container files are recursively parsed to determine whether components of the container files are container or non-container components.
6. The method of claim 4 , wherein deduplicating the plurality of files comprises generating a plurality of filemaps corresponding to the plurality of files.
7. The method of claim 4 , wherein deduplicating the plurality of files comprises generating a plurality of datastore suitcases.
8. The method of claim 7 , wherein the datastore suitcase further comprises a plurality of reference counts corresponding to the plurality of deduplicated data segments.
9. The method of claim 8 , wherein the plurality of deduplicated data segments are determined to be infrequently accessed if they have low associated reference counts.
10. A system, comprising:
an interface configured to receive a plurality of data files;
a processor configured to partition the plurality of data files into a plurality of data segments for deduplication, wherein a segment size is determined using file type information associated with the plurality of data files;
storage configured to maintain a plurality of deduplicated data segments;
wherein a plurality of deduplicated data segments are compressed using a common compression context, wherein the plurality of deduplicated data segments are compressed upon determining that the plurality of deduplicated data segments are infrequently accessed, wherein a compressor examines patterns across the plurality of infrequently accessed deduplicated data segments to determine frequently occurring patterns, wherein a single deduplicated data segment is compressed using a non-shared compression context upon determining that the single deduplicated data segment is accessed frequently.
11. The system of claim 10 , wherein the plurality of data segments span a plurality of files.
12. The system of claim 10 , wherein the plurality of files are determined to be container or non-container files based on the file types associated with the files.
13. The system of claim 12 , wherein non-container files are evaluated to identify whether the non-container files would benefit from more fine-grained segmentation, wherein non-container files that would not benefit from more fine-grained segmentation have segment boundaries set to the file boundaries.
14. The system of claim 13 , wherein container files are recursively parsed to determine whether components of the container files are container or non-container components.
15. The system of claim 13 , wherein deduplicating the plurality of files comprises generating a plurality of filemaps corresponding to the plurality of files.
16. The system of claim 13 , wherein deduplicating the plurality of files comprises generating a plurality of datastore suitcases.
17. The system of claim 16 , wherein the datastore suitcase further comprises a plurality of reference counts corresponding to the plurality of deduplicated data segments.
18. The system of claim 17 , wherein the plurality of deduplicated data segments are determined to be infrequently accessed if they have low associated reference counts.