Method and system for compression aware chunking of backup streams
One example method includes reading a data chunk from a data stream, compressing the data chunk, and calculating a chunk delta. When the chunk delta is greater than zero, the compressed data chunk is appended to an incomplete data chunk. When the chunk delta is zero or less, the boundaries of a completed and compressed data chunk having a size at least as large as a minimum size are declared.
1. A method, comprising:
reading a data chunk from a data stream;
compressing the data chunk;
calculating a chunk delta;
when the chunk delta is greater than zero, appending the compressed data chunk to an incomplete data chunk, and the compressed data chunk is appended to the incomplete chunk even when a size of the compressed data chunk is larger than the chunk delta; and
when the chunk delta is not greater than zero, declaring boundaries of a completed and compressed data chunk having a size at least as large as a minimum size.
2. The method as recited in claim 1 , further comprising setting the minimum chunk size.
3. The method as recited in claim 1 , wherein the completed and compressed data chunk includes an uncompressed data chunk.
4. The method as recited in claim 1 , wherein the chunk delta is a difference between the minimum size and a sum of a size of the compressed data chunk and a size of the incomplete data chunk.
5. The method as recited in claim 1 , wherein a size of the incomplete data chunk is greater than or equal to zero.
6. The method as recited in claim 1 , wherein calculation of the chunk delta is based in part on a size of the compressed data chunk.
7. The method as recited in claim 1 , further comprising deduplicating the completed and compressed data chunk.
8. The method as recited in claim 1 , wherein the method is repeated until the chunk delta is not greater than zero.
9. The method as recited in claim 1 , wherein a location of one of the boundaries is determined by a size of a final data chunk appended before the boundaries were declared.
10. The method as recited in claim 1 , wherein an amount of data read from the data stream is based on a chunk delta value.
11. A non-transitory storage medium having stored therein computer-executable instructions which are executable by one or more hardware processors to perform operations comprising:
reading a data chunk from a data stream;
compressing the data chunk;
calculating a chunk delta;
when the chunk delta is greater than zero, appending the compressed data chunk to an incomplete data chunk, and the compressed data chunk is appended to the incomplete chunk even when a size of the compressed data chunk is larger than the chunk delta; and
when the chunk delta is not greater than zero, declaring boundaries of a completed and compressed data chunk having a size at least as large as a minimum size.
12. The non-transitory storage medium as recited in claim 11 , wherein the operations further comprise setting the minimum chunk size.
13. The non-transitory storage medium as recited in claim 11 , wherein the completed and compressed data chunk includes an uncompressed data chunk.
14. The non-transitory storage medium as recited in claim 11 , wherein the chunk delta is a difference between the minimum size and a sum of a size of the compressed data chunk and a size of the incomplete data chunk.
15. The non-transitory storage medium as recited in claim 11 , wherein a size of the incomplete data chunk is greater than or equal to zero.
16. The non-transitory storage medium as recited in claim 11 , wherein calculation of the chunk delta is based in part on a size of the compressed data chunk.
17. The non-transitory storage medium as recited in claim 11 , wherein the operations further comprise deduplicating the completed and compressed data chunk.
18. The non-transitory storage medium as recited in claim 11 , wherein the operations are repeated until the chunk delta is not greater than zero.
19. The non-transitory storage medium as recited in claim 11 , wherein a location of one of the boundaries is determined by a size of a final data chunk appended before the boundaries were declared.
20. The non-transitory storage medium as recited in claim 11 , wherein an amount of data read from the data stream is based on a chunk delta value.