Object deduplication and application aware snapshots
View Patent ↗Embodiments deploy delayering techniques, and the relationships between successive versions of a rich-media file become apparent. With this, modified rich-media files suddenly present far smaller storage overhead as compared to traditional application-unaware snapshot and versioning implementations. Optimized file data is stored in suitcases. As a file is versioned, each new version of the file is placed in the same suitcase as the previous version, allowing embodiments to employ correlation techniques to enhance optimization savings.
1. A method comprising:
parsing a file to identify boundaries for a plurality of first level objects, including a first compound object, in order to determine whether each of the plurality of first level objects is compound or primitive, wherein a primitive object is the most basic representation of a discrete data structure in the file;
recursively parsing the first compound object until a plurality of lowest level primitive objects is identified;
correlating primitive objects within the file and across a plurality of files;
storing the identified boundaries for use in deduplication;
setting deduplication boundaries at boundaries of the plurality of lowest level primitive objects such that deduplication uses variable sized blocks instead of fixed sized blocks; and
optimizing the plurality of lowest level primitive objects with a plurality of object specific optimization algorithms.
2. The method of claim 1 , wherein object based deduplication is applied to the plurality of lowest level primitive objects.
3. The method of claim 1 , wherein compound objects include zip archives and PDF files.
4. The method of claim 1 , wherein primitive object include JPEG images, TIFF images, and CAD drawings.
5. The method of claim 1 , wherein deduplication is applied to the plurality of lowest level primitive objects.
6. The method of claim 1 , wherein the first compound object comprises a plurality of second level objects.
7. The method of claim 1 , wherein metadata for the plurality of lowest level primitive objects is stored in a separate file.
8. A system comprising:
a processor; and
memory comprising one or more programs, the one or more programs containing instructions for:
parsing a file to identify boundaries for a plurality of first level objects, including a first compound object, in order to determine whether each of the plurality of first level objects is compound or primitive, wherein a primitive object is the most basic representation of a discrete data structure in the file;
recursively parsing the first compound object until a plurality of lowest level primitive objects is identified;
correlating primitive objects within the file and across a plurality of files;
storing the identified boundaries for use in deduplication;
setting deduplication boundaries at boundaries of the plurality of lowest level primitive objects such that deduplication uses variable sized blocks instead of fixed sized blocks; and
optimizing the plurality of lowest level primitive objects with a plurality of object specific optimization algorithms.
9. The system of claim 8 , wherein object based deduplication is applied to the plurality of lowest level primitive objects.
10. The system of claim 8 , wherein compound objects include zip archives and PDF files.
11. The system of claim 8 , wherein primitive object include JPEG images, TIFF images, and CAD drawings.
12. The system of claim 8 , wherein deduplication is applied to the plurality of lowest level primitive objects.
13. The system of claim 8 , wherein the first compound object comprises a plurality of second level objects.
14. The system of claim 8 , wherein metadata for the plurality of lowest level primitive objects is stored in a separate file.
15. A non-transitory computer readable medium comprising one or more programs, the one or more programs containing instructions for:
parsing a file to identify boundaries for a plurality of first level objects, including a first compound object, in order to determine whether each of the plurality of first level objects is compound or primitive, wherein a primitive object is the most basic representation of a discrete data structure in the file;
recursively parsing the first compound object until a plurality of lowest level primitive objects is identified;
correlating primitive objects within the file and across a plurality of files;
storing the identified boundaries for use in deduplication;
setting deduplication boundaries at boundaries of the plurality of lowest level primitive objects such that deduplication uses variable sized blocks instead of fixed sized blocks; and
optimizing the plurality of lowest level primitive objects with a plurality of object specific optimization algorithms.
16. The non-transitory computer readable medium of claim 15 , wherein object based deduplication is applied to the plurality of lowest level primitive objects.
17. The non-transitory computer readable medium of claim 15 , wherein compound objects include zip archives and PDF files.
18. The non-transitory computer readable medium of claim 15 , wherein primitive object include JPEG images, TIFF images, and CAD drawings.
19. The non-transitory computer readable medium of claim 15 , wherein deduplication is applied to the plurality of lowest level primitive objects.
20. The non-transitory computer readable medium of claim 15 , wherein the first compound object comprises a plurality of second level objects.