Efficient compression of data in storage systems through offloading computation to storage devices

1. An apparatus comprising:

in an information processing system implementing data deduplication and compression, wherein the information processing system comprises a set of data storage devices;

at least one of the data storage devices comprising a processing device configured:

to receive, from the information processing system, an instruction to perform at least a portion of a compression operation;

to perform the portion of the compression operation in response to the instruction; and

to send a result of the performed portion of the compression operation to the information processing system;

the processing device being further configured to execute at least one of the following processes:

(i) wherein when the instruction comprises new data to be written in a compressed format, the portion of the compression operation performed by the processing device of the at least one data storage device comprises:

performing standalone compression of the new data;

performing differential compression of the new data with reference to existing data stored on the at least one storage device; and

storing one of the standalone compressed new data and the differential compressed new data based at least in part on a comparison of a first compression ratio of the standalone compressed new data and a second compression ratio of the differential compressed new data; or

(ii) wherein when the instruction comprises new data to be written and a similarity score associated with the new data:

performing the portion of the compression operation comprises comparing the similarity score associated with the new data to one or more similarity scores associated with existing data stored on the at least one storage device; and

sending the result of the performed portion of the compression operation to the information processing system comprises providing a measure of similarity between the similarity score associated with the new data and the one or more similarity scores associated with the existing data stored on the at least one storage device.

2. The apparatus of claim 1 wherein, when the processing device executes process (i), performing the differential compression of the new data comprises utilizing a compression algorithm which allows the compressed new data to have one or more references to the existing data.

3. The apparatus of claim 1 wherein, when the processing device executes process (ii), the processing device is further configured:

to receive, from the information processing system, an additional instruction to store the new data on the at least one storage device, the additional instruction being based at least in part on the measured similarity between the similarity score associated with the new data and the one or more similarity sores associated with the existing data stored on the at least one storage device;

to perform an additional portion of the compression operation in response to the additional instruction, the additional portion of the compression operation comprising compressing the new data and storing the compressed new data on the at least one storage device.

4. The apparatus of claim 1 wherein, when the processing device executes process (ii), the similarity score associated with the new data is determined using a rolling hash with a designated window size.

5. The apparatus of claim 4 wherein the rolling hash comprises a sliding window Rabin hash.

6. The apparatus of claim 1 wherein, when the processing device executes process (ii), at least one of a host device of the information processing system and the processing device of the at least one data storage device is further configured to maintain, in a memory of the at least one data storage device, a cache of similarity scores associated with the existing data stored on the at least one data storage device.

7. The apparatus of claim 6 wherein the cache comprises a least recently used (LRU) cache configured to maintain similarity scores for a designated threshold number of existing data items most recently stored on the at least one data storage device.

8. The apparatus of claim 6 wherein the cache is configured to maintain similarity scores for a designated threshold number of frequently accessed existing data items stored on the at least one data storage device.

9. The apparatus of claim 1 wherein, when the processing device executes process (ii), the processing device is further configured:

to maintain, in a memory of the at least one data storage device, dependency data for differential compressed existing data stored on the at least one data storage device; and

to utilize the dependency data to determine, when receiving requests to overwrite existing data stored on the at least one storage device using differential compression data, whether to overwrite the differential compression data.

10. The apparatus of claim 9 wherein determining whether to overwrite the differential compression data comprises, for a given existing data item stored using the differential compression data:

determining, utilizing the dependency data, a number of other existing data items that reference the differential compression data for the given existing data item;

if the number of other existing data items that reference the differential compression data for the given existing data item is below a designated threshold, decompressing the other existing data items that reference the differential compression data and recompressing the other existing data items that reference the differential compression data by performing standalone compression of the other existing data items.

11. The apparatus of claim 1 wherein the set of data storage devices comprise solid state drives (SSDs).

12. The apparatus of claim 11 wherein the processing device associated with the at least one data storage device comprises one or more of a central processing unit and a hardware accelerator internal to the SSDs.

13. A method comprising:

in an information processing system implementing data deduplication and compression, wherein the information processing system comprises a set of data storage devices, receiving by at least one of the data storage devices comprising a processing device an instruction from the information processing system to perform at least a portion of a compression operation;

performing the portion of the compression operation in response to the instruction; and

sending a result of the performed portion of the compression operation to the information processing system;

the method further comprising at least one of the following processes:

(i) wherein when the instruction comprises new data to be written in a compressed format, the portion of the compression operation performed by the processing device of the at least one data storage device comprises:

performing standalone compression of the new data;

performing differential compression of the new data with reference to existing data stored on the at least one storage device; and

storing one of the standalone compressed new data and the differential compressed new data based at least in part on a comparison of a first compression ratio of the standalone compressed new data and a second compression ratio of the differential compressed new data; or

(ii) wherein when the instruction comprises new data to be written and a similarity score associated with the new data:

performing the portion of the compression operation comprises comparing the similarity score associated with the new data to one or more similarity scores associated with existing data stored on the at least one storage device; and

sending the result of the performed portion of the compression operation to the information processing system comprises providing a measure of similarity between the similarity score associated with the new data and the one or more similarity scores associated with the existing data stored on the at least one storage device.

14. The method of claim 13 , wherein, when process (ii) is executed, the method further comprises:

receiving, from the information processing system, an additional instruction to store the new data on the at least one storage device, the additional instruction being based at least in part on the measured similarity between the similarity score associated with the new data and the one or more similarity sores associated with the existing data stored on the at least one storage device;

performing an additional portion of the compression operation in response to the additional instruction, the additional portion of the compression operation comprising compressing the new data and storing the compressed new data on the at least one storage device.

15. The method of claim 13 , wherein, when process (ii) is executed, at least one of a host device of the information processing system and the processing device of the at least one data storage device is further configured to maintain, in a memory of the at least one data storage device, a cache of similarity scores associated with the existing data stored on the at least one data storage device.

16. The method of claim 13 , wherein, when process (ii) is executed, the method further comprises:

maintaining, in a memory of the at least one data storage device, dependency data for differential compressed existing data stored on the at least one data storage device; and

utilizing the dependency data to determine, when receiving requests to overwrite existing data stored on the at least one storage device using differential compression data, whether to overwrite the differential compression data.

17. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device of at least one data storage device causes said at least one processing device:

in an information processing system implementing data deduplication and compression, wherein the information processing system comprises a set of data storage devices including the at least one data storage device, to receive by the at least one data storage device an instruction from the information processing system to perform at least a portion of a compression operation;

to perform the portion of the compression operation in response to the instruction; and

to send a result of the performed portion of the compression operation to the information processing system;

the program code being further configured to cause the at least one processing device to perform at least one of the following processes:

(i) wherein when the instruction comprises new data to be written in a compressed format, the portion of the compression operation performed by the processing device of the at least one data storage device comprises:

performing standalone compression of the new data;

performing differential compression of the new data with reference to existing data stored on the at least one storage device; and

storing one of the standalone compressed new data and the differential compressed new data based at least in part on a comparison of a first compression ratio of the standalone compressed new data and a second compression ratio of the differential compressed new data; or

(ii) wherein when the instruction comprises new data to be written and a similarity score associated with the new data:

performing the portion of the compression operation comprises comparing the similarity score associated with the new data to one or more similarity scores associated with existing data stored on the at least one storage device; and

sending the result of the performed portion of the compression operation to the information processing system comprises providing a measure of similarity between the similarity score associated with the new data and the one or more similarity scores associated with the existing data stored on the at least one storage device.

18. The computer program product of claim 17 wherein, when process (ii) is executed, the at least one processing device is further configured:

to receive, from the information processing system, an additional instruction to store the new data on the at least one storage device, the additional instruction being based at least in part on the measured similarity between the similarity score associated with the new data and the one or more similarity sores associated with the existing data stored on the at least one storage device;

to perform an additional portion of the compression operation in response to the additional instruction, the additional portion of the compression operation comprising compressing the new data and storing the compressed new data on the at least one storage device.

19. The computer program product of claim 17 wherein, when process (ii) is executed, at least one of a host device of the information processing system and the processing device of the at least one data storage device is further configured to maintain, in a memory of the at least one data storage device, a cache of similarity scores associated with the existing data stored on the at least one data storage device.

20. The computer program product of claim 17 wherein, when process (ii) is executed, the at least one processing device is further configured:

to maintain, in a memory of the at least one data storage device, dependency data for differential compressed existing data stored on the at least one data storage device; and

to utilize the dependency data to determine, when receiving requests to overwrite existing data stored on the at least one storage device using differential compression data, whether to overwrite the differential compression data.