Efficient data management through compressed data interfaces
A system and method for efficiently managing data through compression interfaces may include receiving, by a controller, data, generating, by the controller, a compressed payload based on the data, generating, by the controller, metadata describing the compressed payload, the metadata including fixed size metadata and variable size metadata, generating, by the controller, a data container comprising the uncompressed payload and the metadata, and transmitting, by the controller, the data container to an application. Some implementations of the system may include a storage media, and a storage controller executable by a processor that may include an interface processor, a controller logic, and a media processor configured to communicate with an application and the storage media to perform aspects of the method.
1. A system comprising:
a storage controller logic of a storage device configured to:
receive a compress buffer request from an application, the compress buffer request including uncompressed data;
responsive to receiving the compress buffer request, compress the uncompressed data to generate a compressed payload;
generate a data container having a data container format and including the compressed payload;
determine attributes of a fixed size metadata;
determine attributes of a variable size metadata;
attach, as metadata to the data container, the fixed size metadata and the variable size metadata; and
return the data container having the data container format and including the compressed payload, the fixed size metadata, and the variable size metadata to the application.
2. The system of claim 1 , wherein the storage controller logic is further configured to:
encode the uncompressed data into the compressed payload using data deduplication.
3. The system of claim 2 , wherein encoding the uncompressed data into the compressed payload includes:
processing a data block of the uncompressed data, the data block comprising a plurality of pieces of data;
determining whether a second piece of data of the plurality of pieces of data has a threshold level of similarity to a first piece of data of the plurality of pieces of data; and
encoding the data block based on the determination that the second piece of data has the threshold level of similarity to the first piece of data.
4. The system of claim 1 , wherein:
the fixed size metadata includes data that is opaque to the application; and
the variable size metadata includes data interpretable by the application.
5. The system of claim 1 , wherein the fixed size metadata includes data that is interpretable by the application to allow the application to interpret the variable size metadata.
6. The system of claim 1 , wherein:
the metadata includes validity check information for verifying an integrity of the compressed payload; and
the validity check information for verifying the integrity of the compressed payload is configured to be interpreted by the application.
7. The system of claim 1 , wherein:
the metadata includes validity check information for verifying an integrity of an uncompressed form of the compressed payload; and
the validity check information for verifying the integrity of the uncompressed form of the compressed payload is configured to be opaque to the application.
8. The system of claim 1 , wherein the metadata is configured to allow the application to interact with the compressed payload without decompressing data of the compressed payload.
9. The system of claim 1 , wherein the storage controller logic is further configured to:
receive a decompress buffer request from the application, the decompress buffer request including a second data container and the second data container including a second compressed payload;
decompress the second compressed payload to generate uncompressed data; and
return the uncompressed data to the application.
10. The system of claim 1 , wherein the application verifies integrity of the compressed payload in the data container using the metadata.
11. A method comprising:
receiving, by a storage controller logic of a storage device, a compress buffer request from an application, the compress buffer request including uncompressed data;
responsive to receiving the compress buffer request, compressing, by the storage controller logic of the storage device, the uncompressed data to generate a compressed payload;
generating, by the storage controller logic of the storage device, a data container having a data container format and including the compressed payload;
determining, by the storage controller logic of the storage device, attributes of a fixed size metadata;
determining, by the storage controller logic of the storage device, attributes of a variable size metadata;
attaching, by the storage controller logic of the storage device, the fixed size metadata and the variable size metadata as metadata to the data container; and
returning, by the storage controller logic of the storage device, the data container having the data container format and including the compressed payload, the fixed size metadata, and the variable size metadata to the application.
12. The method of claim 11 , further comprising:
encoding, by the storage controller logic of the storage device, the uncompressed data into the compressed payload using data deduplication.
13. The method of claim 12 , wherein encoding the uncompressed data into the compressed payload includes:
processing a data block of the uncompressed data, the data block comprising a plurality of pieces of data;
determining whether a second piece of data of the plurality of pieces of data has a threshold level of similarity to a first piece of data of the plurality of pieces of data; and
encoding the data block based on the determination that the second piece of data has the threshold level of similarity to the first piece of data.
14. The method of claim 11 , wherein:
the fixed size metadata includes data that is opaque to the application; and
the variable size metadata includes data interpretable by the application.
15. The method of claim 11 , wherein the fixed size metadata includes data that is interpretable by the application to allow the application to interpret the variable size metadata.
16. The method of claim 11 , wherein:
the metadata includes validity check information for verifying an integrity of the compressed payload; and
the validity check information for verifying the integrity of the compressed payload is configured to be interpreted by the application.
17. The method of claim 11 , wherein:
the metadata includes validity check information for verifying an integrity of an uncompressed form of the compressed payload; and
the validity check information for verifying the integrity of the uncompressed form of the compressed payload is configured to be opaque to the application.
18. The method of claim 11 , wherein the metadata is configured to allow the application to interact with the compressed payload without decompressing data of the compressed payload.
19. The method of claim 11 , further comprising:
receiving, by the storage controller logic of the storage device, a decompress buffer request from the application, the decompress buffer request including a second data container and the second data container including a second compressed payload;
decompressing, by the storage controller logic of the storage device, the second compressed payload to generate uncompressed data; and
returning, by the storage controller logic of the storage device, the uncompressed data to the application.
20. A system comprising:
means for receiving a compress buffer request from an application, the compress buffer request including uncompressed data;
means for compressing, responsive to receiving the compress buffer request, the uncompressed data to generate a compressed payload;
means for generating a data container having a data container format and including the compressed payload;
means for determining attributes of a fixed size metadata;
means for determining attributes of a variable size metadata;
means for attaching, as metadata to the data container, the fixed size metadata and the variable size metadata; and
means for returning the data container having the data container format and including the compressed payload, the fixed size metadata, and the variable size metadata to the application.