Application aware deduplication
Deduplication of files or other data or objects in a manner that is aware of a format of the file and the application. A file may be chunked based on the format. The chunks are more consistent and lead to higher deduplication ratios. The file may be presented as a single file. However, the file is stored in chunks or subfiles and deduplication is performed with respect to the chunks. When the file is read, the file is rebuilt from its respective chunks. The files may also be compressed using differential compression, which leverages the content of similar files to compress a current file.
1. A method comprising:
receiving a file to store in a storage of a file system, wherein the file has a format associated with an application;
determining the format of the file, wherein the file includes subfiles;
chunking the file based on the format of the file, wherein each of the chunks corresponds to one or more of the subfiles of the file and wherein chunking the file includes chunking the files at locations that correspond to ends/beginnings of the subfiles;
deduplicating the chunks with respect to other chunks in the storage;
compressing the file using dual compression by first compressing the chunks and then compressing all of the chunks into a compressed file.
2. The method of claim 1 , wherein some of the compressed chunks in the compressed file are pointers.
3. The method of claim 1 , further comprising rebuilding the file when the file is read by decompressing the compressed file and then decompressing the compressed chunks.
4. The method of claim 1 , wherein the chunks are compressed before being de-duplicated such that the deduplication occurs with respect to compressed chunks.
5. The method of claim 1 , wherein the file is a word processing file, a spreadsheet file, an image file, a video file, a presentation file, an email, a database, or combination thereof.
6. The method of claim 1 , further wherein compressing the file includes performing differential compression on the file.
7. The method of claim 6 , wherein differential compression includes replacing portions of the file with references to other files.
8. The method of claim 1 , wherein at least some of the chunks have different sizes.
9. The method of claim 1 , wherein the received file is compressed and chunked while compressed.
10. A non-transitory computer readable medium comprising computer executable instructions that, when executed by a processor, perform operations comprising:
receiving a file to store in a storage of a file system, wherein the file has a format associated with an application;
determining the format of the file, wherein the file includes subfiles;
chunking the file based on the format of the file, wherein each of the chunks corresponds to one or more of the subfiles of the file and wherein chunking the file includes chunking the files at locations that correspond to ends/beginnings of the subfiles;
deduplicating the chunks with respect to other chunks in the storage;
compressing the file using dual compression by first compressing the chunks and then compressing all of the chunks into a compressed file.
11. The non-transitory computer readable medium of claim 10 , wherein some of the compressed chunks in the compressed file are pointers.
12. The non-transitory computer readable medium of claim 10 , further comprising rebuilding the file when the file is read by decompressing the compressed file and then decompressing the compressed chunks.
13. The non-transitory computer readable medium of claim 10 , wherein the chunks are compressed before being de-duplicated such that the deduplication occurs with respect to compressed chunks.
14. The non-transitory computer readable medium of claim 10 , wherein the file is a word processing file, a spreadsheet file, an image file, a video file, a presentation file, an email, a database, or combination thereof.
15. The non-transitory computer readable medium of claim 10 , further wherein compressing the file includes performing differential compression on the file.
16. The non-transitory computer readable medium of claim 15 , wherein differential compression includes replacing portions of the file with references to other files.
17. The non-transitory computer readable medium of claim 10 , wherein at least some of the chunks have different sizes.
18. The non-transitory computer readable medium of claim 10 , wherein the received file is compressed and chunked while compressed.