Data compression in a file storage system
View Patent ↗Embodiments of the present invention provide for compression of files in a filesystem. In embodiments of the present invention, a file is logically partitioned into chunks and each chunk is compressed, such that a request involving a portion of a file can be satisfied by reading and decompressing only the compressed chunk(s) corresponding to that portion.
1. A method for storing a file by a file server in a file storage system, wherein the file is logically partitioned into chunks, the method comprising:
compressing each chunk to form a corresponding compressed chunk;
forming a compressed file including the compressed chunks;
storing the compressed file in the file storage system; and
maintaining mapping metadata for the compressed file, the mapping metadata mapping portions of the original file to corresponding compressed chunks in the compressed file, wherein compressing a chunk to form a corresponding compressed chunk includes generating a header for the compressed chunk, and wherein maintaining the mapping metadata comprises maintaining mapping metadata in the header including maintaining within the header a pointer to a sequentially next compressed chunk in the compressed file.
2. A method according to claim 1 , further comprising:
dividing a non-compressed file into chunks.
3. A method according to claim 2 , wherein dividing the non-compressed file into chunks comprises:
dividing the non-compressed file into nominally fixed-size chunks.
4. A method according to claim 1 , further comprising:
receiving write requests for the file, wherein each write request is treated as a separate chunk.
5. A method according to claim 1 , wherein forming the compressed file comprises:
maintaining mapping metadata within the compressed file.
6. A method according to claim 5 , wherein maintaining mapping metadata within the compressed file comprises maintaining at least one map within the compressed file.
7. A method according to claim 5 , wherein the compressed file includes a root node, and wherein maintaining the mapping metadata within the compressed file includes maintaining mapping metadata within the root node.
8. A method according to claim 1 , wherein maintaining mapping metadata comprises at least one of:
mapping each chunk to its corresponding compressed chunk; and
mapping fixed-size ranges of the file to corresponding compressed chunks in the compressed file.
9. A method according to claim 1 , further comprising maintaining compression metadata for the compressed file, the compression metadata including at least one of:
a compression type for the compressed file;
a compression type for each compressed chunk of the compressed file;
a size for each chunk;
a size for each compressed chunk; and
an indicator for each compressed chunk indicating whether the compressed chunk contains compressed or uncompressed data.
10. A method for storing a file by a file server in a file storage system, wherein the file is logically partitioned into chunks, the method comprising:
compressing each chunk to form a corresponding compressed chunk;
forming a compressed file including the compressed chunks;
storing the compressed file in the file storage system; and
maintaining mapping metadata for the compressed file, the mapping metadata mapping portions of the original file to corresponding compressed chunks in the compressed file, wherein maintaining the mapping metadata comprises maintaining mapping metadata in a map file separate from the compressed file and maintaining a reference to the map file within the compressed file, and wherein storing a reference to the map file within the compressed file includes storing the reference within a root node of the compressed file.
11. A method according to claim 10 , further comprising at least one of:
dividing a non-compressed file into chunks; and
receiving write requests for the file, wherein each write request is treated as a separate chunk.
12. A method according to claim 10 , wherein maintaining mapping metadata comprises mapping each chunk to its corresponding compressed chunk.
13. A method according to claim 10 , wherein maintaining mapping metadata comprises mapping fixed-size ranges of the file to corresponding compressed chunks in the compressed file.
14. A method according to claim 10 , further comprising maintaining compression metadata for the compressed file, the compression metadata including at least one of:
a compression type for the compressed file;
a compression type for each compressed chunk of the compressed file;
a size for each chunk;
a size for each compressed chunk; and
an indicator for each compressed chunk indicating whether the compressed chunk contains compressed or uncompressed data.
15. A method for storing a file by a file server in a file storage system, wherein the file is logically partitioned into chunks, the method comprising:
compressing each chunk to form a corresponding compressed chunk;
forming a compressed file including the compressed chunks;
storing the compressed file in the file storage system;
maintaining mapping metadata for the compressed file, the mapping metadata mapping portions of the original file to corresponding compressed chunks in the compressed file; and
upon receiving a request relating to a portion of the original file:
identifying at least one compressed chunk associated with the portion of the original file using the map information;
retrieving each identified compressed chunk from the compressed file;
decompressing each retrieved compressed chunk sufficiently to recover original data associated with the portion; and
satisfying the requesting using the recovered original data, and when satisfying the request results in a modified chunk:
compressing the modified chunk to form a modified compressed chunk;
storing the modified compressed chunk as part of the compressed file such that when the size of the modified compressed chunk is less than or equal to the size of the corresponding compressed chunk retrieved from the compressed file, inserting the modified compressed chunk into the compressed file in place of the compressed chunk, and when the size of the modified compressed chunk is greater than the size of the corresponding compressed chunk retrieved from the compressed file, inserting the modified compressed chunk elsewhere in the compressed file; and
updating the mapping metadata to include the modified compressed chunk.
16. A method according to claim 15 , wherein decompressing each retrieved compressed chunk sufficiently to recover original data associated with the portion comprises terminating decompression of a compressed chunk upon recovering a sufficient amount of original data for satisfying the request.
17. A method according to claim 15 , further comprising at least one of:
dividing a non-compressed file into chunks; and
receiving write requests for the file, wherein each write request is treated as a separate chunk.
18. A method according to claim 17 , wherein forming the compressed file comprises maintaining mapping metadata in the compressed file.
19. A method according to claim 15 , further comprising recovering unused space within the compressed file.
20. A method according to claim 19 , wherein recovering unused space within the compressed file comprises:
decompressing the compressed file; and
recompressing the compressed file.
21. A method according to claim 15 , wherein maintaining mapping metadata comprises at least one of:
mapping each chunk to its corresponding compressed chunk; and
mapping fixed-size ranges of the file to corresponding compressed chunks in the compressed file.
22. A method according to claim 15 , further comprising maintaining compression metadata for the compressed file, the compression metadata including at least one of:
a compression type for the compressed file;
a compression type for each compressed chunk of the compressed file;
a size for each chunk;
a size for each compressed chunk; and
an indicator for each compressed chunk indicating whether the compressed chunk contains compressed or uncompressed data.