System method for I/O acceleration in hybrid storage wherein copies of data segments are deleted if identified segments does not meet quality level threshold

1. A method for maintaining data in a storage system, the system comprising:

organizing a plurality of storage devices of different types into a plurality of distinct groups based on their respective technology type and performance;

selecting one or more groups associated with low random input/output operations per second (IOPS) and high throughput performance for a storage pool tier;

selecting one or more groups associated with high random IOPS and low throughput performance for an input/output (I/O) accelerator tier;

identifying a plurality of data segments to be copied from memory that are already stored in the storage pool tier, wherein the identified data segments include statistics usable to assign a quality level for each data segment;

evaluating the identified data segments for the plurality of accelerator tiers, wherein the evaluation is based on the quality level for a particular evaluated data segment, and the plurality of accelerator tiers each have their own respective quality level threshold;

maintaining copies of the identified data segments in the plurality of accelerator tiers based on the evaluations that meet the quality level threshold for the accelerator tiers; and

deleting copies of the identified data segments in the plurality of accelerator tiers based on the evaluations that do not meet the quality level threshold for the accelerator tiers.

2. The method of claim 1 , further comprising maintaining the redundancy level, for at least one accelerator tier, matching or exceeding the redundancy level of the storage pool for data segments not in the storage pool.

3. The method of claim 1 , further comprising where write request data is copied to write back accelerator tier by detecting predetermined I/O parameters, directing small (relative and predetermined) random write requests to groups that can handle higher IOPS but lower throughput and directing large (relative and predetermined) write requests to groups that can handle relatively higher throughput but relatively lower IOPS.

4. The method of claim 3 , wherein consecutive write requests already in a write-request queue are combined prior to directing a write request.

5. The method of claim 1 , further comprising:

dynamically calculating and storing a plurality of parameters for each data segment in metadata that persist across storage system reboots and failovers; and

determining a data quality level for each of the data segments based on the plurality of parameters.

6. The method of claim 5 , wherein the data quality level comprises a level of relevance to spatial locality and a temporal locality.

7. The method of claim 5 , further comprising populating an accelerator tier with data segments based on data quality level obtained.

8. The method of claim 7 , wherein populating an accelerator tier includes:

retaining I/O request data in memory of the storage system after I/O is complete;

maintaining the data uncompressed if the quality level exceeds a first threshold;

attempting to compress the data if the quality level is below the first threshold but exceeds a second threshold;

maintaining compressed copy if the compression attempt is successful;

maintaining the data uncompressed if the compression attempt is not successful; and

attempting to copy the data into an accelerator tier if the quality level is below the second threshold but exceeds a third threshold.

9. The method of claim 5 , wherein determining the quality level includes:

decreasing the quality level if an access to a contiguous data segment occurs sequentially during a given time frame; and

increasing the quality level if access to two or more small data segments located in nearby logical block range occurs randomly during a given timeframe.

10. A method for improving I/O performance, the method comprising:

organizing a plurality of storage devices of different types into a plurality of distinct groups based on their respective technology type and performance;

selecting one or more groups of a first type of storage devices as a storage pool, wherein the first type of storage devices have low random input/output operations per second (IOPS) and high throughput performance;

selecting one or more groups of a second type of storage devices as one or more accelerator tiers for the storage pool, wherein the second type of storage devices are distinct from the first type of storage devices, wherein the at first storage type chosen based on one or more characteristics preferable to the second storage type, and wherein the second storage type is chosen based on characteristics preferable to the first storage type;

receiving a write request to write data into one or more storage devices, wherein the received write request has an associated size;

writing a plurality of larger sized write requests to the storage pool;

writing a plurality of smaller sized write requests to the one or more accelerator tiers;

generating a write request to transfer data stored in an accelerator tier to the storage pool, wherein the generated write request comprises of data read from the accelerator tier, wherein the generated write request optimizes a number of program erase cycles associated with the accelerator tiers, and wherein the generated write request is triggered when a total size of all data written to a particular accelerator tier exceeds a predefined threshold; and

writing the data read from the accelerator tiers, via the generated write request, to the storage pool, wherein the generated write request includes one or more write requests that includes two or more smaller sized read requests from the accelerator tiers.

11. The method of claim 10 , wherein the one or more first characteristics include one or more of monetary cost, capacity, data recovery, reliability on power loss, commercial availability of desired storage size and failure rate.

12. The method of claim 10 , wherein the one or more second characteristics comprise one or more of a random IOPS, sequential IOPS, and capacity.

13. A non-transitory computer readable storage medium having embodied thereon a program, the program being executable by a processor to perform a method for maintaining data in a storage system, the method comprising:

organizing a plurality of storage devices of different types into a plurality of distinct groups based on their respective technology type and performance;

selecting one or more groups associated with low random input/output operations per second (IOPS) and high throughput performance for a storage pool tier;

selecting one or more groups with high random IOPS but lower throughput performance for input/output (I/O) accelerator tier;

identifying a plurality of data segments to be copied from memory that are already stored in the storage pool tier, wherein the identified data segments include statistics usable to assign a quality level for each data segment;

evaluating the identified data segments for the plurality of accelerator tiers, wherein the evaluation is based on the quality level for a particular evaluated data segment, and the plurality of accelerator tiers each have their own respective quality level threshold;

maintaining copies of the identified data segments in the plurality of accelerator tiers based on the evaluations that meet the quality level threshold for the accelerator tiers; and

deleting copies of the identified data segments in the plurality of accelerator tiers based on the evaluations that do not meet the quality level threshold for the accelerator tiers.

14. The non-transitory computer readable storage medium of claim 13 , further comprising maintaining the redundancy level, for at least one accelerator tier, matching or exceeding the redundancy level of the storage pool for data segments not in the storage pool.

15. The non-transitory computer readable storage medium of claim 13 , further comprising where write request data is copied to write back accelerator tier by detecting predetermined I/O parameters, directing small (relative and predetermined) random write requests to groups that can handle higher IOPS but lower throughput and directing large (relative and predetermined) write requests to groups that can handle relatively higher throughput but relatively lower IOPS.

16. The non-transitory computer readable storage medium of claim 15 , wherein consecutive write requests already in a write-request queue are combined prior to directing a write request.

17. The non-transitory computer readable storage medium of claim 13 , further comprising:

dynamically calculating and storing a plurality of parameters for each data segment in metadata that persist across storage system reboots and failovers; and

determining a data quality level for each of the data segments based on the plurality of parameters.

18. The non-transitory computer readable storage medium of claim 17 , wherein the data quality level comprises a level of relevance to spatial locality and a temporal locality.

19. The non-transitory computer readable storage medium of claim 17 , further comprising populating an accelerator tier with data segments based on data quality level obtained.

20. The non-transitory computer readable storage medium of claim 19 , wherein populating an accelerator tier includes:

retaining I/O request data in memory of the storage system after I/O is complete;

maintaining the data uncompressed if the quality level exceeds a first threshold;

attempting to compress the data if the quality level is below the first threshold but exceeds a second threshold;

maintaining compressed copy if the compression attempt is successful;

maintaining the data uncompressed if the compression attempt is not successful; and

attempting to copy the data into an accelerator tier if the quality level is below the second threshold but exceeds a third threshold.

21. The non-transitory computer readable storage medium of claim 17 , wherein determining the quality level includes:

decreasing the quality level if an access to a contiguous data segment occurs sequentially during a given time frame; and

increasing the quality level if access to two or more small data segments located in nearby logical block range occurs randomly during a given timeframe.

22. A non-transitory computer readable storage medium having embodied thereon a program, the program being executable by a processor to perform a method for improving I/O performance, the method comprising:

organizing a plurality of storage devices of different types into a plurality of distinct groups based on their respective technology type and performance;

selecting one or more groups of a first type of storage devices as a storage pool, wherein the first type of storage devices have low random input/output operations per second (IOPS) and high throughput performance;

selecting one or more groups of a second type of storage devices as one or more accelerator tiers for the storage pool, wherein the second type of storage devices are distinct from the first type of storage devices, wherein the at first storage type chosen based on one or more characteristics preferable to the second storage type, and wherein the second storage type is chosen based on characteristics preferable to the first storage type;

receiving a write request to write data into one or more storage devices, wherein the received write request has an associated size;

writing a plurality of larger sized write requests to the storage pool;

writing a plurality of smaller sized write requests to the one or more accelerator tiers;

generating a write request to transfer data stored in an accelerator tier to the storage pool, wherein the generated write request comprises of data read from the accelerator tier, wherein the generated write request optimizes a number of program erase cycles associated with the accelerator tiers, and wherein the generated write request is triggered when a total size of all data written to a particular accelerator tier exceeds a predefined threshold; and

writing the data read from the accelerator tiers, via the generated write request, to the storage pool, wherein the generated write request includes one or more write requests that includes two or more smaller sized read requests from the accelerator tiers.

23. The non-transitory computer readable storage medium of claim 22 , wherein the one or more first characteristics include one or more of monetary cost, capacity, data recovery, reliability on power loss, commercial availability of desired storage size and failure rate.

24. The non-transitory computer readable storage medium of claim 22 , wherein the one or more second characteristics comprise one or more of a random IOPS, sequential IOPS, and capacity.

25. A computer system, comprising:

memory;

one or more processors; and

an application stored in memory and executable by the one or more processors to:

organize a plurality of storage devices of different types into a plurality of distinct groups based on their respective technology type and performance,

select one or more groups associated with low random input/output operations per second (IOPS) and high throughput performance for a storage pool tier,

select one or more groups associated with high random IOPS and low throughput performance for an input/output (I/O) accelerator tier,

identify a plurality of data segments to be copied from memory that are already stored in the storage pool tier, wherein the identified data segments include statistics usable to assign a quality level for each data segment

evaluate the identified data segments for the plurality of accelerator tiers, wherein the evaluation is based on the quality level for a particular evaluated data segment, and the plurality of accelerator tiers each have their own respective quality level threshold,

maintain copies of the identified data segments in the plurality of accelerator tiers based on the evaluations that meet the quality level threshold for the accelerator tiers, and

delete copies of the identified data segments in the plurality of accelerator tiers based on the evaluations that do not meet the quality level threshold for the accelerator tiers.

26. The computer system of claim 25 , further comprising maintaining the redundancy level, for at least one accelerator tier, matching or exceeding the redundancy level of the storage pool for data segments not in the storage pool.

27. The computer system of claim 25 , wherein the application further executable to copy write request data to a write back accelerator tier by detecting predetermined I/O parameters, directing small (relative and predetermined) random write requests to groups that can handle higher IOPS but lower throughput and directing large (relative and predetermined) write requests to groups that can handle relatively higher throughput but relatively lower IOPS.

28. The computer system of claim 27 , wherein consecutive write requests already in a write-request queue are combined prior to directing a write request.

29. The computer system of claim 25 , wherein the application further executable to:

dynamically calculate and store a plurality of parameters for each data segment in metadata that persist across storage system reboots and failovers, and

determine a data quality level for each of the data segments based on the plurality of parameters.

30. The computer system of claim 29 , wherein the data quality level comprises a level of relevance to spatial locality and a temporal locality.

31. The computer system of claim 29 , wherein the application further executable to populate an accelerator tier with data segments based on data quality level obtained.

32. The computer system of claim 31 , wherein the application further executable to populate accelerator tier by:

retaining I/O request data in memory of the storage system after I/O is complete;

maintaining the data uncompressed if the quality level exceeds a first threshold;

attempting to compress the data if the quality level is below the first threshold but exceeds a second threshold;

maintaining compressed copy if the compression attempt is successful;

maintaining the data uncompressed if the compression attempt is not successful; and

attempting to copy the data into an accelerator tier if the quality level is below the second threshold but exceeds a third threshold.

33. The computer system of claim 29 , wherein determining the quality level includes:

decreasing the quality level if an access to a contiguous data segment occurs sequentially during a given time frame; and

increasing the quality level if access to two or more small data segments located in nearby logical block range occurs randomly during a given timeframe.

34. A computer system, comprising:

memory;

one or more processors; and

an application stored in memory, and executable by the one or more processors to:

organize a plurality of storage devices of different types into a plurality of distinct groups based on their respective technology type and performance,

select one or more groups of a first type of storage devices as a storage pool, wherein the first type of storage devices have low random input/output operations per second (IOPS) and high throughput performance,

select one or more groups of a second type of storage devices as one or more accelerator tiers for the storage pool, wherein the second type of storage devices are distinct from the first type of storage devices, wherein the at first storage type chosen based on one or more characteristics preferable to the second storage type, and wherein the second storage type is chosen based on characteristics preferable to the first storage type,

receive a write request to write data into one or more storage devices, wherein the received write request has an associated size, wherein a plurality of larger sized write requests are written to the storage pool, and wherein a plurality of smaller sized write requests are written to the one or more accelerator tiers,

generate a write request to transfer data stored in an accelerator tier to the storage pool, wherein the generated write requests comprises of the data read from the accelerator tier, wherein the generated write request optimizes a number of program erase cycles associated with the accelerator tiers, and wherein the generated write request is triggered when a total size of all data written to a particular accelerator tier exceeds a predefined threshold, and

write the data read from the accelerator tiers, via the generated write request, to the storage pool, wherein the generated write request includes one or more write requests that includes two or more smaller sized read requests from the accelerator tiers.

35. The computer system of claim 34 , wherein the one or more first characteristics include one or more of monetary cost, capacity, data recovery, reliability on power loss, commercial availability of desired storage size and failure rate.

36. The computer system of claim 34 , wherein the one or more second characteristics comprise one or more of a random input/output operations per second (IOPS), sequential IOPS, and capacity.

37. The computer system of claim 25 further comprising a plurality of storage devices to provide the storage pool.

38. The computer system of claim 34 further comprising a plurality of storage devices to provide the storage pool.