System and methods for in-storage on-demand data decompression
A system and methods for in-storage on-demand data decompression. Compressed data are stored in a storage device connected to a host computer. When decompressed data are needed, the host computer sends a decompression command to the storage device indicating which data are to be decompressed, and instructing it how to decompress the data. The storage device decompresses the data and stores the decompressed data, making it available to the host.
1. A storage device, comprising:
a storage interface, configured to be connected to a host computer,
a digital circuit; and
persistent storage media,
the digital circuit being configured to:
receive, from the host computer, through the storage interface, a decompression command comprising:
an identifier, identifying a quantity of compressed data; and
decompression instructions for decompressing the compressed data;
retrieve the quantity of compressed data from the storage media; and
decompress the compressed data, according to the decompression instructions,
wherein the storage interface is NVMe, and the decompression command is part of an NVMe command, the NVMe command having an address selected from among a range of addresses reserved for transmission of decompression commands to the storage device.
2. The storage device of claim 1 , wherein the digital circuit is further configured to:
receive, from the host computer, through the storage interface, a location identifier, identifying a destination location where the decompressed data are to be stored; and
store the decompressed data at the destination location.
3. The storage device of claim 1 , wherein the decompression command is part of a payload of a transmission control protocol/internet protocol (TCP/IP) packet, the TCP/IP packet being part of the payload of the NVMe command.
4. The storage device of claim 1 , wherein the digital circuit is further configured to send a completion message to the host computer, after the compressed data have been decompressed.
5. The storage device of claim 1 , wherein the storage device is configured as an object store, and the quantity of data is an object.
6. The storage device of claim 1 , wherein the digital circuit is a processing circuit configured to decompress the compressed data by directly executing the decompression instructions.
7. The storage device of claim 1 , wherein the digital circuit is a processing circuit configured to translate the decompression instructions into directly executable instructions, and to decompress the compressed data by directly executing the directly executable instructions.
8. The storage device of claim 1 , wherein the digital circuit is a programmable digital circuit, and the decompression instructions are a logic program.
9. The storage device of claim 1 , wherein the digital circuit is a digital signal processor having an instruction word length greater than 32 bits.
10. The storage device of claim 1 , wherein the digital circuit comprises a hardware accelerator configured to execute a plurality of functions, each of the functions being a component of one or more of a plurality of decompression methods.
11. The storage device of claim 1 , wherein the digital circuit comprises:
a first hardware decompressor configured to perform decompression according to a first decompression method; and
a second hardware decompressor configured to perform decompression according to a second decompression method, and
wherein the decompression instructions include an identifier identifying one member of a list of decompression methods.
12. A method for operating a storage device comprising persistent storage media and connected, through a storage interface, to a host computer, the method comprising:
receiving, by the storage device, from the host computer, through the storage interface, a decompression command comprising:
an identifier, identifying a quantity of compressed data; and
decompression instructions for decompressing the compressed data;
retrieving the quantity of compressed data from the storage media; and
decompressing the compressed data, according to the decompression instructions,
wherein the storage interface is Non Volatile Memory Express (NVMe), and the decompression command is part of an NVMe command, the NVMe command having an address selected from among a range of addresses reserved for transmission of decompression commands to the storage device.
13. The method of claim 12 , further comprising:
receiving, from the host computer, through the storage interface, a location identifier, identifying a destination location where the decompressed data are to be stored; and
storing the decompressed data at the destination location.
14. The method of claim 12 , wherein the decompressing of the compressed data, according to the decompression instructions, comprises decompressing of the compressed data by directly executing the decompression instructions.
15. The method of claim 12 , wherein the decompressing of the compressed data, according to the decompression instructions, comprises translating the decompression instructions into directly executable instructions, and decompressing of the compressed data by directly executing the directly executable instructions.
16. The method of claim 12 , wherein the decompressing of the compressed data, according to the decompression instructions, comprises configuring a programmable digital circuit according to the decompression instructions, and decompressing of the compressed data by the programmable digital circuit.
17. The method of claim 12 , wherein the decompression command is part of a payload of a transmission control protocol/internet protocol (TCP/IP) packet, the TCP/IP packet being part of the payload of the NVMe command.