Techniques for determining compression tiers and using collected compression hints
Tiers of compression algorithms may be determined using compression information collected regarding compression ratios achieved for data sets using compression algorithms. Each tier may meet specified criteria regarding expected compression ratios achieved for a specified portion or number of data sets. Compression algorithms of each tier may be implemented by a different hardware device that may include hardware accelerators for the algorithms of the tier. Different tiers, and thus different hardware devices, achieve different levels of compression. A recommendation may be provided using compression information collected, such as from one of the hosts, regarding which hardware device to use for compression. The recommendation may be to purchase a license to use or whether to purchase a particular hardware device for compression. Compression information may be collected by a host that issues tagged I/Os providing a hint regarding what compression algorithm to use for the particular I/O operation data.
1. A method of determining and using a plurality of tiers of compression algorithms comprising:
receiving a plurality of sets of compression information for a plurality of compression algorithms used to compress a plurality of data sets, wherein each of the plurality of sets of compression information is collected for I/O operations issued to a different one of a plurality of data storage systems;
determining, using the plurality of sets of compression information, a first of the plurality of tiers of compression algorithms, wherein each compression algorithm of the plurality of compression algorithms in the first tier achieves first estimated compression ratio criteria for at least a first portion of the plurality of data sets, wherein compression algorithms in the first tier are implemented using a first hardware (HW) device, wherein the first estimated compression ratio criteria for the first tier indicates that each compression algorithm of the first tier, for at least a first minimum percentage of data compressed using said each compression algorithm, achieves a specified compression ratio within a first specified compression ratio range;
determining, using the plurality of sets of compression information, a second of the plurality of tiers of compression algorithms, wherein each compression algorithm of the plurality of compression algorithms in the second tier achieves second estimated compression ratio criteria for at least a second portion of the plurality of data sets, wherein each compression algorithm in the second tier is expected to achieve a higher rate of compression than compression algorithms in the first tier, wherein compression algorithms in the second tier are implemented using a second HW device, wherein the second estimated compression ratio criteria for the second tier indicates that each compression algorithm of the second tier, for at least a second minimum percentage of data compressed using said each compression algorithm, achieves a specified compression ratio within a second specified compression ratio range which is different than the first specified compression ratio range;
providing a recommendation, in accordance with one of the plurality of sets of compression information regarding I/O operations issued to a first of the plurality of data storage systems and in accordance with first licensing information of the first data storage system, regarding whether to use the first HW device or the second HW device in connection with compression on the first data storage system, wherein the first licensing information includes per data set licensing information for a first data set of the plurality of data sets and a second data set of the plurality of data sets, wherein the first licensing information indicates that the first HW device but not the second HW device is licensed for use with the first data set, and wherein the first licensing information indicates that both the first HW device and the second HW device are licensed for use with the second data set;
compressing, based on the provided recommendation and the first licensing information, the first data set using the first HW device but not the second HW device, wherein the compression of the first data set achieves the first estimated compression ratio criteria;
compressing, based on the provided recommendation and the first licensing information, the second data set using a selected one of the first HW device and the second HW device, wherein the compression of the second data set achieves the first estimated compression ratio criteria when the first HW device is selected and the compression of the second data set achieves the second estimated compression ratio criteria when the second HW device is selected; and
wherein the first HW device and the second HW device are included in the first data storage system, wherein the first data storage system comprises one or more processors, wherein the first HW device includes one or more first processors used to perform compression and decompression processing for compression algorithms of the first tier, wherein the second HW device includes one or more second processors used to perform compression and decompression processing for compression algorithms of the second tier.
2. The method of claim 1 , wherein a first set of the plurality of sets of compression information is collected by a host in connection with I/Os directed to at least some of the plurality of data sets stored on the first data storage system, and wherein the first set of compression information is sent from the host to a centralized collection site.
3. The method of claim 2 , wherein the first set of compression information is sent to the first data storage system, and then from the first data storage system to the centralized collection site.
4. The method of claim 1 , wherein a first set of the plurality of sets of compression information is collected by the first data storage system in connection with I/Os directed to at least some of the plurality of data sets stored on the first data storage system, and wherein the first set of compression information is sent from the first data storage system to a centralized collection site.
5. The method of claim 1 , wherein a first set of the plurality of sets of compression information is collected by a host in connection with I/Os directed to at least some of the plurality of data sets stored on the first data storage system, and wherein the method includes:
receiving the first set of compression information from the host at the first data storage system;
receiving, from the host at the first data storage system, a first write I/O operation that writes first data and is directed to the first data set stored on the first data storage system;
selecting, in accordance with the first set of compression information and the first licensing information, a particular one of the plurality of compression algorithms included in the first tier to use to compress the first data stored in the first data set; and
compressing the first data using the first HW device implementing the particular one of the plurality of compression algorithms selected.
6. The method of claim 1 , wherein a first set of the plurality of sets of compression information is collected by a host in connection with I/Os directed to at least some of the plurality of data sets stored on the first data storage system, and wherein the method includes:
receiving, on the host from an application, a first write I/O operation that writes first data and is directed to the first data set stored on the first data storage system;
selecting, by the host and in accordance with the first set of compression information, a particular one of the plurality of compression algorithms included in the first tier to use to compress the first data stored in the first data set;
tagging the first write I/O operation with a tag identifying the particular one of the plurality of compression algorithms;
sending the first write I/O operation with the tag from the host to the first data storage system; and
performing first processing on the first data storage system to determine which of the plurality of compression algorithms to use to compress the first data stored in the first data set.
7. The method of claim 6 , wherein the first processing uses the first licensing information for the first data set identifying whether each of the first HW device and the second HW device is licensed for use by the first data set.
8. The method of claim 1 , wherein each of the plurality of data sets includes any of: one or more logical devices, one or more file-based entities, data used by one or more hosts, a portion of a logical device, a plurality of portions of a plurality of logical devices, and data used by one or more applications.
9. The method of claim 1 , wherein the first HW device and the second HW device are used to offload processing from the one or more processors of the first data storage system where the one or more processors of the first data storage system are not used to perform compression and decompression processing associated with any compression algorithm of the first tier and also not used to perform compression and decompression processing associated with any compression algorithm of the second tier.
10. A system comprising:
at least one processor; and
a memory comprising code stored thereon that, when executed, performs a method of determining and using a plurality of tiers of compression algorithms comprising:
receiving a plurality of sets of compression information for a plurality of compression algorithms used to compress a plurality of data sets, wherein each of the plurality of sets of compression information is collected for I/O operations issued to a different one of a plurality of data storage systems;
determining, using the plurality of sets of compression information, a first of the plurality of tiers of compression algorithms, wherein each compression algorithm of the plurality of compression algorithms in the first tier achieves first estimated compression ratio criteria for at least a first portion of the plurality of data sets, wherein compression algorithms in the first tier are implemented using a first hardware (HW) device, wherein the first estimated compression ratio criteria for the first tier indicates that each compression algorithm of the first tier, for at least a first minimum percentage of data compressed using said each compression algorithm, achieves a specified compression ratio within a first specified compression ratio range;
determining, using the plurality of sets of compression information, a second of the plurality of tiers of compression algorithms, wherein each compression algorithm of the plurality of compression algorithms in the second tier achieves second estimated compression ratio criteria for at least a second portion of the plurality of data sets, wherein each compression algorithm in the second tier is expected to achieve a higher rate of compression than compression algorithms in the first tier, wherein compression algorithms in the second tier are implemented using a second HW device, wherein the second estimated compression ratio criteria for the second tier indicates that each compression algorithm of the second tier, for at least a second minimum percentage of data compressed using said each compression algorithm, achieves a specified compression ratio within a second specified compression ratio range which is different than the first specified compression ratio range;
providing a recommendation, in accordance with one of the plurality of sets of compression information regarding I/O operations issued to a first of the plurality of data storage systems and in accordance with first licensing information of the first data storage system, regarding whether to use the first HW device or the second HW device in connection with compression on the first data storage system, wherein the first licensing information includes per data set licensing information for a first data set of the plurality of data sets and a second data set of the plurality of data sets, wherein the first licensing information indicates that the first HW device but not the second HW device is licensed for use with the first data set, and wherein the first licensing information indicates that both the first HW device and the second HW device are licensed for use with the second data set;
compressing, based on the provided recommendation and the first licensing information, the first data set using the first HW device but not the second HW device, wherein the compression of the first data set achieves the first estimated compression ratio criteria;
compressing, based on the provided recommendation and the first licensing information, the second data set using a selected one of the first HW device and the second HW device, wherein the compression of the second data set achieves the first estimated compression ratio criteria when the first HW device is selected and the compression of the second data set achieves the second estimated compression ratio criteria when the second HW device is selected; and
wherein the first HW device and the second HW device are included in the first data storage system, wherein the first data storage system comprises one or more processors, wherein the first HW device includes one or more first processors used to perform compression and decompression processing for compression algorithms of the first tier, wherein the second HW device includes one or more second processors used to perform compression and decompression processing for compression algorithms of the second tier.
11. A non-transitory computer readable medium comprising code stored thereon that, when executed, performs a method of determining and using a plurality of tiers of compression algorithms comprising:
receiving a plurality of sets of compression information for a plurality of compression algorithms used to compress a plurality of data sets, wherein each of the plurality of sets of compression information is collected for I/O operations issued to a different one of a plurality of data storage systems;
determining, using the plurality of sets of compression information, a first of the plurality of tiers of compression algorithms, wherein each compression algorithm of the plurality of compression algorithms in the first tier achieves first estimated compression ratio criteria for at least a first portion of the plurality of data sets, wherein compression algorithms in the first tier are implemented using a first hardware (HW) device, wherein the first estimated compression ratio criteria for the first tier indicates that each compression algorithm of the first tier, for at least a first minimum percentage of data compressed using said each compression algorithm, achieves a specified compression ratio within a first specified compression ratio range;
determining, using the plurality of sets of compression information, a second of the plurality of tiers of compression algorithms, wherein each compression algorithm of the plurality of compression algorithms in the second tier achieves second estimated compression ratio criteria for at least a second portion of the plurality of data sets, wherein each compression algorithm in the second tier is expected to achieve a higher rate of compression than compression algorithms in the first tier, wherein compression algorithms in the second tier are implemented using a second HW device, wherein the second estimated compression ratio criteria for the second tier indicates that each compression algorithm of the second tier, for at least a second minimum percentage of data compressed using said each compression algorithm, achieves a specified compression ratio within a second specified compression ratio range which is different than the first specified compression ratio range;
providing a recommendation, in accordance with one of the plurality of sets of compression information regarding I/O operations issued to a first of the plurality of data storage systems and in accordance with first licensing information of the first data storage system, regarding whether to use the first HW device or the second HW device in connection with compression on the first data storage system, wherein the first licensing information includes per data set licensing information for a first data set of the plurality of data sets and a second data set of the plurality of data sets, wherein the first licensing information indicates that the first HW device but not the second HW device is licensed for use with the first data set, and wherein the first licensing information indicates that both the first HW device and the second HW device are licensed for use with the second data set;
compressing, based on the provided recommendation and the first licensing information, the first data set using the first HW device but not the second HW device, wherein the compression of the first data set achieves the first estimated compression ratio criteria;
compressing, based on the provided recommendation and the first licensing information, the second data set using a selected one of the first HW device and the second HW device, wherein the compression of the second data set achieves the first estimated compression ratio criteria when the first HW device is selected and the compression of the second data set achieves the second estimated compression ratio criteria when the second HW device is selected; and
wherein the first HW device and the second HW device are included in the first data storage system, wherein the first data storage system comprises one or more processors, wherein the first HW device includes one or more first processors used to perform compression and decompression processing for compression algorithms of the first tier, wherein the second HW device includes one or more second processors used to perform compression and decompression processing for compression algorithms of the second tier.