Compression and deduplication layered driver
A method, apparatus, and system for interposed file system driver is provided, which provides a logical file system on top of an existing base file system. One such interposed file system driver is a compression and deduplication layered driver (“COLD driver”). File system operations are intercepted from the operating system through the COLD driver, which is provided as an upper-level operating system driver that operates on top of an existing base file system. By processing file data through various modules, the existing base file system can be extended as a logical file system with compression, deduplication, indexing, and other functionality. The COLD driver can be implemented without requiring modifications to existing base file system structures or base file system drivers. Server deployments may thus leverage the additional file system functionality provided by the COLD driver without having to migrate to another file system.
1. A method comprising:
invoking an interposed file system driver to request one or more operations on one or more logical files in a logical file system accessible via said interposed file system driver;
wherein said logical file system is associated with metadata that defines, at least in part, said logical file system, said metadata including:
a metadata mapping between logical files in said logical file system and base files in a base file system, and
compression metadata;
wherein the base file system is mountable by an operating system;
performing the one or more operations, wherein performing the one or more operations includes:
accessing said metadata mapping to determine that one or more base files are mapped to said one or more logical files;
accessing said one or more base files determined to be mapped to said one or more logical files;
translating said one or more base files into a buffer, wherein translating includes decompressing said one or more base files into said buffer; and
wherein the method is performed on one or more computing devices.
2. The method of claim 1 , wherein said metadata is in a file of said base file system.
3. The method of claim 1 , wherein said metadata is separate from a dedicated file system metadata area of said base file system.
4. The method of claim 1 further comprising, prior to invoking said interposed file system driver, creating and storing said metadata in said base file system.
5. The method of claim 1 , wherein said one or more operations requested comprise a write from a buffer into said one or more logical files, and wherein said performing said one or more operations further include:
translating said buffer into said one or more base files according to said metadata.
6. The method of claim 5 , wherein prior to said translating, said metadata mapping maps at least one logical data block in said one or more logical files to a base data block in said one or more base files, and wherein said performing further includes:
updating said metadata mapping such that said at least one logical data block is remapped to a commit container file of said base file system.
7. The method of claim 5 , wherein said translating compresses said buffer into said one or more base files, and wherein said translating adds a reference to said compression metadata in said metadata, said compression metadata usable to decompress said compressed buffer.
8. The method of claim 7 , wherein said compressing said buffer uses adaptive compression to skip compressing blocks of said buffer below a minimum compression ratio.
9. The method of claim 7 , wherein said compressing said buffer applies a policy to skip compressing blocks of said buffer corresponding to any tail base data block of said one or more base files.
10. The method of claim 5 , wherein said translating further calculates a plurality of checksums for a plurality of logical data blocks comprising said one or more logical files, said plurality of checksums written into said compression metadata.
11. The method of claim 10 , wherein said translating further checks said plurality of checksums for duplicate checksums existing in said compression metadata, remapping logical data blocks having duplicate checksums to a single base data block in said base files of said base file system.
12. The method of claim 11 , wherein said checking of said plurality of checksums further verifies said duplicate checksums by a binary compare of said logical data blocks having said duplicate checksums.
13. The method of claim 5 , wherein said translating further indexes said buffer into a content metadata management database, and wherein said content metadata management database is utilized for policy enforcement on said logical file system.
14. The method of claim 1 , wherein decompressing said one or more base files into said buffer includes reading said one or more base files that are mapped within deduplication data, wherein said deduplication data includes a mapping between a plurality of checksums for a plurality of logical data blocks comprising said one or more logical files to one or more base data blocks comprising said one or more base files.
15. The method of claim 1 , wherein said interposed file system driver is limited to a portable system application programming interface (API) for interacting with said base file system.
16. A non-transitory computer-readable medium storing one or more sequences of instructions which, when executed by one or more processors, cause:
invoking an interposed file system driver to request one or more operations on one or more logical files in a logical file system accessible via said interposed file system driver;
wherein said logical file system is associated with metadata that defines, at least in part, said logical file system, said metadata including:
a metadata mapping between logical files in said logical file system and base files in a base file system, and
compression metadata;
wherein the base file system is mountable by an operating system;
performing the one or more operations, wherein performing the one or more operations includes:
accessing said metadata mapping to determine that one or more base files are mapped to said one or more logical files;
accessing said one or more base files determined to be mapped to said one or more logical files; and
translating said one or more base files into a buffer, wherein translating includes decompressing said one or more base files into said buffer.
17. The non-transitory computer-readable medium of claim 16 , wherein said metadata is in a file of said base file system.
18. The non-transitory computer-readable medium of claim 16 , wherein said metadata is separate from a dedicated file system metadata area of said base file system.
19. The non-transitory computer-readable medium of claim 16 , where said one or more sequences of instructions comprise instructions, that when executed by said one or more processors, cause: prior to invoking said interposed file system driver, creating and storing said metadata in said base file system.
20. The non-transitory computer-readable medium of claim 16 , wherein said one or more operations requested comprise a write from a buffer into said one or more logical files, and wherein said performing said one or more operations further include:
translating said buffer into said one or more base files according to said metadata.
21. The non-transitory computer-readable medium of claim 20 , wherein prior to said translating, said metadata mapping maps at least one logical data block in said one or more logical files to a base data block in said one or more base files, and wherein said performing further includes:
updating said metadata mapping such that said at least one logical data block is remapped to a commit container file of said base file system.
22. The non-transitory computer-readable medium of claim 20 , wherein said translating compresses said buffer into said one or more base files, and wherein said translating adds a reference to said compression metadata in said metadata, said compression metadata usable to decompress said compressed buffer.
23. The non-transitory computer-readable medium of claim 22 , wherein said compressing said buffer uses adaptive compression to skip compressing blocks of said buffer below a minimum compression ratio.
24. The non-transitory computer-readable medium of claim 22 , wherein said compressing said buffer applies a policy to skip compressing blocks of said buffer corresponding to any tail base data block of said one or more base files.
25. The non-transitory computer-readable medium of claim 20 , wherein said translating further calculates a plurality of checksums for a plurality of logical data blocks comprising said one or more logical files, said plurality of checksums written into said compression metadata.
26. The non-transitory computer-readable medium of claim 25 , wherein said translating further checks said plurality of checksums for duplicate checksums existing in said compression metadata, remapping logical data blocks having duplicate checksums to a single base data block in said base files of said base file system.
27. The non-transitory computer-readable medium of claim 26 , wherein said checking of said plurality of checksums further verifies said duplicate checksums by a binary compare of said logical data blocks having said duplicate checksums.
28. The non-transitory computer-readable medium of claim 20 , wherein said translating further indexes said buffer into a content metadata management database, and wherein said content metadata management database is utilized for policy enforcement on said logical file system.
29. The non-transitory computer-readable medium of claim 16 , wherein decompressing said one or more base files into said buffer includes reading said one or more base files that are mapped within deduplication data, wherein said deduplication data includes a mapping between a plurality of checksums for a plurality of logical data blocks comprising said one or more logical files to one or more base data blocks comprising said one or more base files.