Data compression with inline compression metadata
Techniques for handling data compression in which metadata that indicates which portions of data are compressed are which portions of data are not compressed are disclosed. Segments of a buffer referred to as block groups store compressed blocks of data along with uncompressed blocks of data and hash blocks. If a block group includes a block that is a hash of another block in the block group, then the other block is considered to be compressed. If the block group does not include a block that is a hash of another block in the block group, then the blocks in the block group are uncompressed. The hash function to generate the hash is selected to prevent “collisions,” which occur when the data being stored in the buffer is such that it is possible for a hash block and an uncompressed block to be the same.
1. A method for accessing compressed data, the method comprising:
receiving an access request at a memory interface, the access request specifying data and an access type, the data including a block group stored in a buffer that stores a plurality of block groups that are each the same size, each block also being the same size, the access request received from a data consumer;
responsive to the access request, performing a signature operation for a block of the block group, the signature operation identifying the block as being either compressed or uncompressed, based on whether the block group includes a block that is a signature of another block in the block group;
transmitting all blocks of the block group to the data consumer if the signature operation identifies the block as uncompressed, and
transmitting only compressed blocks of the block group if the signature operation identifies the block as compressed,
wherein the block groups of the buffer include either all uncompressed blocks or one compressed block and one signature block comprising a signature of the one compressed block.
2. The method of claim 1 , wherein:
the access request comprises a read request.
3. The method of claim 2 , further comprising:
responsive to the read request, fetching read data from the buffer, the read data including the compressed block and the signature block.
4. The method of claim 3 , wherein the signature operation comprises:
performing a signature function on the compressed block to obtain a signature of the compressed block;
determining that the signature block matches the signature of the compressed block; and
discarding the signature block in response to determining that the signature block matches the signature of the compressed block.
5. The method of claim 4 , further comprising:
transmitting the compressed block and the uncompressed block to a compression/decompression unit, with an indication that the compressed block is compressed;
based on the indication, decompressing the compressed block to generate a decompressed block; and
transmitting the decompressed block and the uncompressed block to the data consumer.
6. The method of claim 1 , wherein the signature operation comprises one of:
a hash function, a duplication function, a bitwise inverse duplication function, a parity function, a checksum function, a modular sum function, or a modulo function.
7. A memory system comprising:
a memory; and
a memory interface configured to:
receive an access request specifying data and an access type, the data including a block group stored in a buffer that stores a plurality of block groups that are each the same size, each block also being the same size, the access request received from a data consumer;
responsive to the access request, perform a signature operation for a block of the block group, the signature operation identifying the block as being either compressed or uncompressed;
transmit all blocks of the block group to the data consumer if the signature operation identifies the block as uncompressed, and
transmit only compressed blocks of the block group if the signature operation identifies the block as compressed,
wherein the block groups of the buffer include either all uncompressed blocks or one compressed block and one signature block comprising a signature of the one compressed block.
8. The memory system of claim 7 , wherein:
the access request comprises a read request.
9. The memory system of claim 8 , wherein the memory interface is further configured to:
responsive to the read request, fetch read data from the buffer, the read data including the compressed block and the signature block.
10. The memory system of claim 9 , wherein the memory interface is configured to perform the signature operation by:
performing a signature function on the compressed block to generate a signature of the compressed block;
determining that the signature block matches the signature of the compressed block; and
discarding the signature block in response to determining that the signature block matches the signature of the compressed block.
11. The memory system of claim 10 , wherein:
the memory system further comprises a compression/decompression block
the memory interface is further configured to transmit the compressed block and the uncompressed block to the compression/decompression unit, with an indication that the compressed block is compressed; and
the compression/decompression block is configured to:
based on the indication, decompress the compressed block to generate a decompressed block, and
transmit the decompressed block and the uncompressed block to the data consumer.
12. The memory system of claim 7 , wherein the signature operation comprises one of:
a hash function, a duplication function, a bitwise inverse duplication function, a parity function, a checksum function, a modular sum function, or a modulo function.
13. A computer system comprising:
a data processor;
a compression/decompression unit coupled to the data processor;
a memory interface coupled to the compression/decompression unit via a data conduit and a compression indicator side-channel and to the data processor via a control side-channel; and
a memory coupled to the memory interface,
wherein the memory interface is configured to:
receive an access request specifying data and an access type, the data including a block group stored in a buffer that stores a plurality of block groups that are each the same size, each block also being the same size, the access request received from a data consumer;
responsive to the access request, perform a signature operation for a block of the block group, the signature operation identifying the block as being either compressed or uncompressed;
transmit all blocks of the block group to the data consumer if the signature operation identifies the block as uncompressed, and
transmit only compressed blocks of the block group if the signature operation identifies the block as compressed,
wherein the block groups of the buffer include either all uncompressed blocks or one compressed block and one signature block comprising a signature of the one compressed block.
14. The computer system of claim 13 , wherein the signature operation comprises one of:
a hash function, a duplication function, a bitwise inverse duplication function, a parity function, a checksum function, a modular sum function, or a modulo function.