Using variable sized uncompressed data blocks to address file positions when simultaneously compressing multiple blocks
An apparatus in one embodiment comprises at least one processing device comprising a processor coupled to a memory. The processing device is configured to determine a plurality of block offsets indicating respective positions of data within a plurality of uncompressed data blocks respectively encoded to have variable sizes, to generate a block index file comprising the plurality of block offsets, a plurality of block positions and a plurality of block sizes respectively corresponding to the plurality of uncompressed data blocks, to compress data from at least one uncompressed data block of the plurality of uncompressed data blocks to at least one compressed data block of a plurality of compressed data blocks, and to utilize the block index file in connection with decompressing the compressed data.
1. An apparatus comprising:
at least one processing device comprising a processor coupled to a memory;
the processing device being configured:
to determine a plurality of block offsets indicating respective positions of data within a plurality of uncompressed data blocks respectively encoded to have variable sizes;
to generate a block index file comprising the plurality of block offsets, a plurality of block positions and a plurality of block sizes respectively corresponding to the plurality of uncompressed data blocks;
to sequentially write the data from the plurality of uncompressed data blocks to a plurality of memory buffers corresponding to the plurality of uncompressed data blocks, wherein the plurality of uncompressed data blocks are in an ordered sequence;
to compress data from at least a first uncompressed data block of the plurality of uncompressed data blocks to at least one compressed data block of a plurality of compressed data blocks;
wherein the data from the first uncompressed data block is in a given memory buffer of the plurality of memory buffers prior to being compressed;
during the compression of the data from the first uncompressed data block to the at least one compressed data block, to simultaneously write data from one or more additional uncompressed data blocks of the plurality of uncompressed data blocks to one or more of the plurality of memory buffers;
wherein the one or more additional uncompressed data blocks are subsequent in the ordered sequence relative to the first uncompressed data block; and
to utilize the block index file in connection with decompressing the compressed data.
2. The apparatus of claim 1 wherein the processing device is implemented in a host device configured to communicate over a network with a storage system.
3. The apparatus of claim 1 wherein the processing device is implemented in a storage system configured to communicate over a network with a host device.
4. The apparatus of claim 1 wherein the plurality of block offsets are respectively determined from the start of a corresponding uncompressed data block of the plurality of uncompressed data blocks.
5. The apparatus of claim 1 wherein, for each uncompressed data block of the plurality of uncompressed data blocks, the block index file designates a first number of bytes for a block offset and a block position, and second number of bytes for a block size.
6. The apparatus of claim 5 wherein the first number comprises 8 bytes and the second number comprises 4 bytes.
7. The apparatus of claim 1 wherein each of the plurality of compressed data blocks comprises a block position indicating the start of a given compressed data block and a block offset of the plurality of block offsets.
8. The apparatus of claim 7 wherein the processing device is further configured to access the given compressed data block based on the block position and decompress data from the given compressed data block.
9. The apparatus of claim 7 wherein, in utilizing the block index file in connection with decompressing the compressed data, the processing device is configured:
to determine whether the block offset is within an uncompressed data block corresponding to the given compressed data block; and
to decompress data from a subsequent compressed data block in an ordered sequence relative to the given compressed data block if the block offset is not within the uncompressed data block corresponding to the given compressed data block.
10. The apparatus of claim 1 wherein the block index file is uncompressed.
11. The apparatus of claim 1 wherein the plurality of block offsets respectively correspond to a number of uncompressed bytes in remaining ones of the plurality of uncompressed data blocks waiting to be compressed at a given time.
12. A method comprising:
determining a plurality of block offsets indicating respective positions of data within a plurality of uncompressed data blocks respectively encoded to have variable sizes;
generating a block index file comprising the plurality of block offsets, a plurality of block positions and a plurality of block sizes respectively corresponding to the plurality of uncompressed data blocks;
sequentially writing the data from the plurality of uncompressed data blocks to a plurality of memory buffers corresponding to the plurality of uncompressed data blocks, wherein the plurality of uncompressed data blocks are in an ordered sequence;
compressing data from at least a first uncompressed data block of the plurality of uncompressed data blocks to at least one compressed data block of a plurality of compressed data blocks;
wherein the data from the first uncompressed data block is in a given memory buffer of the plurality of memory buffers prior to being compressed;
during the compression of the data from the first uncompressed data block to the at least one compressed data block, simultaneously writing data from one or more additional uncompressed data blocks of the plurality of uncompressed data blocks to one or more of the plurality of memory buffers;
wherein the one or more additional uncompressed data blocks are subsequent in the ordered sequence relative to the first uncompressed data block; and
utilizing the block index file in connection with decompressing the compressed data;
wherein the method is implemented by at least one processing device comprising a processor coupled to a memory.
13. The method of claim 12 wherein, for each uncompressed data block of the plurality of uncompressed data blocks, the block index file designates a first number of bytes for a block offset and a block position, and second number of bytes for a block size.
14. The method of claim 12 wherein each of the plurality of compressed data blocks comprises a block position indicating the start of a given compressed data block and a block offset of the plurality of block offsets.
15. The method of claim 14 wherein utilizing the block index file in connection with decompressing the compressed data comprises:
determining whether the block offset is within an uncompressed data block corresponding to the given compressed data block; and
decompressing data from a subsequent compressed data block in an ordered sequence relative to the given compressed data block if the block offset is not within the uncompressed data block corresponding to the given compressed data block.
16. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:
to determine a plurality of block offsets indicating respective positions of data within a plurality of uncompressed data blocks respectively encoded to have variable sizes;
to generate a block index file comprising the plurality of block offsets, a plurality of block positions and a plurality of block sizes respectively corresponding to the plurality of uncompressed data blocks;
to sequentially write the data from the plurality of uncompressed data blocks to a plurality of memory buffers corresponding to the plurality of uncompressed data blocks, wherein the plurality of uncompressed data blocks are in an ordered sequence;
to compress data from at least a first uncompressed data block of the plurality of uncompressed data blocks to at least one compressed data block of a plurality of compressed data blocks;
wherein the data from the first uncompressed data block is in a given memory buffer of the plurality of memory buffers prior to being compressed;
during the compression of the data from the first uncompressed data block to the at least one compressed data block, to simultaneously write data from one or more additional uncompressed data blocks of the plurality of uncompressed data blocks to one or more of the plurality of memory buffers;
wherein the one or more additional uncompressed data blocks are subsequent in the ordered sequence relative to the first uncompressed data block; and
to utilize the block index file in connection with decompressing the compressed data.
17. The computer program product of claim 16 wherein each of the plurality of compressed data blocks comprises a block position indicating the start of a given compressed data block and a block offset of the plurality of block offsets.
18. The computer program product of claim 17 wherein, in utilizing the block index file in connection with decompressing the compressed data, the program code when executed by the at least one processing device causes said at least one processing device:
to determine whether the block offset is within an uncompressed data block corresponding to the given compressed data block; and
to decompress data from a subsequent compressed data block in an ordered sequence relative to the given compressed data block if the block offset is not within the uncompressed data block corresponding to the given compressed data block.
19. The computer program product of claim 16 wherein the plurality of block offsets respectively correspond to a number of uncompressed bytes in remaining ones of the plurality of uncompressed data blocks waiting to be compressed at a given time.
20. The method of claim 12 wherein the plurality of block offsets respectively correspond to a number of uncompressed bytes in remaining ones of the plurality of uncompressed data blocks waiting to be compressed at a given time.