Dual layer deduplication for a file system running over a deduplicated block storage
An aspect of dual layer deduplication for a file system running over a deduplication block storage system includes accessing a file by a file system driver of the file system. The file is stored as one of a plurality of files in the file system. A further aspect includes breaking the file down into multiple blocks and generating chunks from the blocks. For at least one of the chunks, a trail of zeros is added until a size of a respective one of the chunks is a multiple of a block size of the deduplication block storage system.
1. A method for dual layer deduplication for a file system running over a deduplication block storage system, comprising:
accessing a file by a file system driver of the file system, the file stored as one of a plurality of files in the file system;
breaking the file down into multiple blocks;
generating chunks from the blocks, wherein for at least one of the chunks adding a trail of zeros until a size of a respective one of the chunks is a multiple of a block size of the deduplication block storage system, and
performing, by the deduplication block storage system, deduplication on the chunks and compression on deduplicated chunks having the trailing zeros;
wherein the file includes data and metadata that describes a structure of the data, the metadata for each file including a list of locations where the file is chunked.
2. The method of claim 1 , wherein the generating chunks is performed by a variable length chunking algorithm.
3. The method of claim 1 , wherein the chunks are one of byte aligned and bit aligned.
4. The method of claim 1 , wherein a chunk size for the files is greater than a chunk size for the block deduplication.
5. The method of claim 1 , wherein each of the blocks has an equal number of characters.
6. A system for dual layer deduplication for a file system running over a deduplication block storage system, comprising:
a memory comprising computer-executable instructions; and
a processor executing the computer-executable instructions, the computer-executable instructions when executed by the processor cause the processor to perform operations comprising:
accessing a file by a file system driver of the file system, the file stored as one of a plurality of files in the file system;
breaking the file down into multiple blocks;
generating chunks from the blocks, wherein for at least one of the chunks adding a trail of zeros until a size of a respective one of the chunks is a multiple of a block size of the deduplication block storage system; and
performing, by the deduplication block storage system, deduplication on the chunks and compression on deduplicated chunks having the trailing zeros;
wherein the file includes data and metadata that describes a structure of the data, the metadata for each file including a list of locations where the file is chunked.
7. The system of claim 6 , wherein the generating chunks is performed by a variable length chunking algorithm.
8. The system of claim 6 , wherein the chunks are one of byte aligned and bit aligned.
9. The system of claim 6 , wherein a chunk size for the files is greater than a chunk size for the block deduplication.
10. The system of claim 6 , wherein each of the blocks has an equal number of characters.
11. A computer program product embodied on a non-transitory computer readable medium, the computer program product including instructions that, when executed by a computer causes the computer to perform operations comprising:
accessing a file by a file system driver of the file system, the file stored as one of a plurality of files in the file system;
breaking the file down into multiple blocks;
generating chunks from the blocks, wherein for at least one of the chunks adding a trail of zeros until a size of a respective one of the chunks is a multiple of a block size of the deduplication block storage system; and
performing, by the deduplication block storage system, deduplication on the chunks and compression on deduplicated chunks having the trailing zeros;
wherein the file includes data and metadata that describes a structure of the data, the metadata for each file including a list of locations where the file is chunked.
12. The computer program product of claim 11 , wherein the generating chunks is performed by a variable length chunking algorithm.
13. The computer program product of claim 11 , wherein the chunks are one of byte aligned and bit aligned.
14. The computer program product of claim 11 , wherein a chunk size for the files is greater than a chunk size for the block deduplication, and wherein each of the blocks has an equal number of characters.