Application aware deduplication
Deduplication of files or other data or objects in a manner that is aware of a format of the file and the application. A file may be chunked based on the format. The chunks are more consistent and lead to higher deduplication ratios. The file may be presented as a single file. However, the file is stored in chunks or subfiles and deduplication is performed with respect to the chunks. When the file is read, the file is rebuilt from its respective chunks. The files may also be compressed using differential compression, which leverages the content of similar files to compress a current file.
1. A method for performing deduplication in a file system, the method comprising:
receiving a file from a client to store in a storage of a file system, wherein the file has a format associated with an application;
determining the format of the file, wherein the file includes subfiles;
chunking the file based on the format of the file, wherein at least one of the chunks corresponds to more than one of the subfiles of the file; and
deduplicating the chunks with respect to other chunks stored in the storage.
2. The method of claim 1 , further comprising chunking the file at locations that correspond to ends/beginnings of the subfiles.
3. The method of claim 1 , further comprising presenting the file as a single file to the client and storing the file as chunks in the storage.
4. The method of claim 3 , wherein the chunks are stored in a compressed form.
5. The method of claim 3 , wherein the chunks are stored in an uncompressed form.
6. The method of claim 1 , further comprising rebuilding the file when the file is read from chunks corresponding to the file in the storage.
7. The method of claim 1 , further comprising performing differential compression on the file by compressing the file with respect to a reference file that has similar properties.
8. The method of claim 1 , further comprising decompressing the file before chunking the file.
9. A non-transitory computer readable medium comprising instructions that, when executed, perform the method of claim 1 .
10. An appliance configured to perform deduplication on files, the appliance comprising:
a storage; and
a deduplication engine comprising a processor, wherein the deduplication engine is configured to:
receive a file from a client to store in the storage of the appliance, wherein the file has a format associated with an application;
determine the format of the file, wherein the file includes subfiles;
chunk the file based on the format of the file, wherein at least one of the chunks corresponds to more than one subfiles of the file; and
deduplicate the chunks with respect to other chunks stored in the storage.
11. The appliance of claim 10 , wherein the deduplication engine is configured to chunk the file at locations that correspond to ends/beginnings of the subfiles.
12. The appliance of claim 10 , wherein the deduplication engine is configured to present the file as a single file to the client and store the file as chunks in the storage.
13. The appliance of claim 10 , wherein the deduplication engine is configured to store the chunks in a compressed form and/or in an uncompressed form.
14. The appliance of claim 10 , wherein the deduplication engine is configured to rebuild the file when the file is read by reading chunks corresponding to the file in the storage, wherein the chunks are identified from an index maintained by the deduplication engine.
15. The appliance of claim 10 , wherein the deduplication engine is configured to perform differential compression on the file by compressing the file with respect to a reference file that has similar properties.
16. The appliance of claim 10 , wherein the deduplication engine is configured to decompress the file before chunking the file.
17. The appliance of claim 15 , wherein the file, when differentially compressed, includes references to a reference file, wherein the similar properties includes at least one of a same location or a same name.
18. The appliance of claim 17 , wherein the deduplication engine is configured to rebuild the differentially compressed file by decompressing the reference file and the differentially compressed file.
19. A method comprising:
receiving a file from a client to store in a storage of a file system, wherein the file has a format associated with an application;
determining the format of the file, wherein the file includes subfiles; chunking the file based on the format of the file, wherein each of the chunks correspond to one or more subfiles of the file; and
deduplicating the chunks with respect to other chunks stored in the storage, differentially compressing the file with respect to a reference file that has similar properties, wherein the file, when differentially compressed, includes references to a reference file, and
rebuilding the differentially compressed file by decompressing the reference file and the differentially compressed file.
20. A non-transitory computer readable medium comprising instructions that, when executed, perform the method of claim 19 .