Object deduplication and application aware snapshots
View Patent ↗Embodiments deploy delayering techniques, and the relationships between successive versions of a rich-media file become apparent. With this, modified rich-media files suddenly present far smaller storage overhead as compared to traditional application-unaware snapshot and versioning implementations. Optimized file data is stored in suitcases. As a file is versioned, each new version of the file is placed in the same suitcase as the previous version, allowing embodiments to employ correlation techniques to enhance optimization savings.
1. A method of reducing data duplication in a computer system storing files, the method comprising:
(a) parsing a file to determine native application boundaries of applications within the file;
(b) determining if the applications are compressed;
(c) decompressing the applications within the file with a technique appropriate for each application, wherein a first technique used for decompressing a first application is different from a second technique used for decompressing a second application;
(d) recursively performing steps (a)-(d) until all primitive objects are uncovered and decompressed;
(e) correlating primitive objects within each file and across a plurality of files accessible by the computer system; and
(f) reducing redundancy between two or more primitive objects within related or versioned files, wherein reducing redundancy comprises creating a suitcase file configured to hold metadata and compressed data associated with a plurality of files on a network attached storage system.
2. The method of claim 1 , wherein reducing redundancy further comprises storing metadata for each of the plurality of files redundantly in the suitcase file, wherein commit bits are stored with the metadata.
3. The method of claim 2 , wherein reducing redundancy further comprises writing compressed data corresponding to each of the plurality files in the suitcase file on the network attached storage system.
4. The method of claim 3 , wherein reducing redundancy further comprises storing index information in the suitcase file, the index information allowing access to each of the plurality of files in the suitcase file.
5. The method of claim 1 , wherein an index number in a file name is used to access index information in the suitcase file to allow access to corresponding compressed data in the suitcase file.
6. The method of claim 1 , wherein the metadata is stored twice and surrounded with commit bits.
7. The method of claim 1 , wherein index information comprises a next_index and a next_offset.
8. The method of claim 1 , wherein index information further comprises data_length and data_info fields for each of the plurality of files.
9. The method of claim 1 , wherein a file name index number is included in a hard link to the plurality of files in the suitcase file.
10. The method of claim 1 , wherein a simulated link is mapped to a hard link.
11. The method of claim 10 , wherein the simulated link to a compressed file has a name matching the name of the file in precompressed form.
12. A method of managing a data storage system, comprising:
determining if a file is of a compound nature containing underlying primitive objects or of a primitive nature;
extracting the underlying primitive objects from files determined to be compound, at least one extraction comprising two layers of decoding with two different codecs;
correlating the extracted objects with the information represented by the extracted objects; and
reducing information redundancy between two or more primitive objects within related or versioned files, wherein reducing redundancy comprises creating a suitcase file configured to hold metadata and compressed data, wherein reducing redundancy comprises creating a suitcase file configured to hold metadata and compressed data associated with a plurality of files on a network attached storage system.
13. The method of claim 12 , wherein reducing redundancy further comprises writing compressed data corresponding to each of the plurality files in the suitcase file on the network attached storage system.
14. The method of claim 13 , wherein reducing redundancy further comprises storing index information in the suitcase file, the index information allowing access to each of the plurality of files in the suitcase file.
15. The method of claim 12 , wherein an index number in a file name is used to access index information in the suitcase file to allow access to corresponding compressed data in the suitcase file.
16. A non-transitory computer readable medium, comprising:
computer code for determining if a file is of a compound nature containing underlying primitive objects or of a primitive nature;
computer code for extracting the underlying primitive objects from files determined to be compound, at least one extraction comprising two layers of decoding with two different codecs;
computer code for correlating the extracted objects with the information represented by the extracted objects; and
computer code for reducing information redundancy between two or more primitive objects within related or versioned files, wherein reducing redundancy comprises creating a suitcase file configured to hold metadata and compressed data, wherein reducing redundancy comprises creating a suitcase file configured to hold metadata and compressed data associated with a plurality of files on a network attached storage system.
17. The non-transitory computer readable medium of claim 16 , wherein reducing redundancy further comprises writing compressed data corresponding to each of the plurality files in the suitcase file on the network attached storage system.