System and method for managing cleaning policies of storage devices in storage device pools using self-monitored statistics and input/output statistics
A method for managing a plurality of storage devices includes obtaining, by a storage device cleaning manager, a set of self-monitored statistics, performing an initial concern analysis to generate an initial concern prediction for each of the plurality of storage devices in a storage system, wherein the set of self-monitored statistics are associated with the plurality of storage devices, updating a cleaning policy based on the initial concern prediction, obtaining input/output (I/O) statistics, after updating the cleaning policy based on the initial concern prediction, performing a secondary concern analysis using the I/O statistics to generate a secondary concern prediction for each of the plurality of storage devices, wherein the I/O statistics are associated with the plurality of storage devices, further updating the cleaning policy, and performing a cleaning of at least a portion of the plurality of storage devices based on the updated cleaning policy.
1. A method for managing a plurality of storage devices, the method comprising:
obtaining, by a storage device cleaning manager, a set of self-monitored statistics;
performing an initial concern analysis using the set of self-monitored statistics to generate an initial concern prediction for each of the plurality of storage devices in a storage system, wherein the set of self-monitored statistics are associated with the plurality of storage devices;
updating a cleaning policy for a second plurality of storage devices based on the initial concern prediction,
wherein the cleaning policy specifies a schedule for cleaning data in the second plurality of storage devices, and
wherein the plurality of storage devices comprises the second plurality of storage devices;
obtaining input/output (I/O) statistics;
after updating the cleaning policy based on the initial concern prediction, performing a secondary concern analysis using the I/O statistics to generate a secondary concern prediction for each of the plurality of storage devices, wherein the I/O statistics are associated with the plurality of storage devices;
further updating the cleaning policy based on the secondary concern prediction to obtain an updated cleaning policy; and
performing a cleaning of the second plurality of storage devices based on the updated cleaning policy.
2. The method of claim 1 , further comprising:
sending a concern predictions diagnostics report to an administrative system.
3. The method of claim 1 , wherein the initial concern analysis comprises performing a classification analysis on the set of self-monitored statistics.
4. The method of claim 3 , wherein a portion of the set of self-monitored statistics is provided to the storage device cleaning manager by a storage device monitoring agent associated with a storage device pool of the storage device pools.
5. The method of claim 1 , wherein the plurality of the storage devices are grouped into storage device pools.
6. The method of claim 1 , wherein the set of self-monitored statistics comprise at least one of: power usage, number of hours in which a storage device in the plurality of storage devices is powered on, a number of times the storage device is powered on, and an average amount of data storage in the storage device over a period of time.
7. The method of claim 1 , wherein the I/O statistics comprise at least one of: a read rate at a specified period of time, a write rate at the specified period of time, a bit error rate at the specified period of time, and a latency spike in the read rate or write rate.
8. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for managing a plurality of storage devices, the method comprising:
obtaining, by a storage device cleaning manager, a set of self-monitored statistics;
performing an initial concern analysis using the set of self-monitored statistics to generate an initial concern prediction for each of the plurality of storage devices in a storage system, wherein the set of self-monitored statistics are associated with the plurality of storage devices;
updating a cleaning policy for a second plurality of storage devices based on the initial concern prediction,
wherein the cleaning policy specifies a schedule for cleaning data in the second plurality of storage devices, and
wherein the plurality of storage devices comprises the second plurality of storage devices;
obtaining input/output (I/O) statistics;
after updating the cleaning policy based on the initial concern prediction, performing a secondary concern analysis using the I/O statistics to generate a secondary concern prediction for each of the plurality of storage devices, wherein the I/O statistics are associated with the plurality of storage devices;
further updating the cleaning policy based on the secondary concern prediction to obtain an updated cleaning policy; and
performing a cleaning of the second plurality of storage devices based on the updated cleaning policy.
9. The non-transitory computer readable medium of claim 8 , the method further comprising:
sending a concern predictions diagnostics report to an administrative system.
10. The non-transitory computer readable medium of claim 8 , wherein the initial concern analysis comprises performing a classification analysis on the set of self-monitored statistics.
11. The non-transitory computer readable medium of claim 8 , wherein the plurality of the storage devices are grouped into storage device pools.
12. The non-transitory computer readable medium of claim 11 , wherein a portion of the set of self-monitored statistics are provided to the storage device cleaning manager by a storage device monitoring agent associated with a storage device pool of the storage device pools.
13. The non-transitory computer readable medium of claim 8 , wherein the set of self-monitored statistics comprise at least one of: a power usage, a number of hours in which a storage device in the plurality of storage devices is powered on, a second number of times the storage device is powered on, and an average amount of data storage in the storage device over a period of time.
14. The non-transitory computer readable medium of claim 8 , wherein the I/O statistics comprise at least one of: a read rate at a specified period of time, a write rate at the specified period of time, a bit error rate at the specified period of time, and a latency spike in the read rate or write rate.
15. A system, comprising:
a processor; and
memory comprising instructions which, when executed by the processor, perform a method, the method comprising:
obtaining, by a storage device cleaning manager, a set of self-monitored statistics;
performing an initial concern analysis using the set of self-monitored statistics to generate an initial concern prediction for each of a plurality of storage devices in a storage system, wherein the set of self-monitored statistics are associated with the plurality of storage devices;
updating a cleaning policy for a second plurality of storage devices based on the initial concern prediction,
wherein the cleaning policy specifies a schedule for cleaning data in the second plurality of storage devices, and
wherein the plurality of storage devices comprises the second plurality of storage devices;
obtaining input/output (I/O) statistics;
after updating the cleaning policy based on the initial concern prediction, performing a secondary concern analysis using the I/O statistics to generate a secondary concern prediction for each of the plurality of storage devices,
wherein the I/O statistics are associated with the plurality of storage devices;
further updating the cleaning policy based on the secondary concern prediction to obtain an updated cleaning policy; and
performing a cleaning of the second plurality of storage devices based on the updated cleaning policy.
16. The system of claim 15 , the method further comprising:
sending a concern predictions diagnostics report to an administrative system.
17. The system of claim 15 , wherein the initial concern analysis comprises performing a classification analysis on the set of self-monitored statistics.
18. The system of claim 17 , wherein a portion of the set of self-monitored statistics are provided to the storage device cleaning manager by a storage device monitoring agent associated with a storage device pool of the storage device pools.
19. The system of claim 15 , wherein the plurality of the storage devices are grouped into storage device pools.
20. The system of claim 15 ,
wherein the set of self-monitored statistics comprise at least one of: a power usage, a number of hours in which a storage device in the plurality of storage devices is powered on, a second number of times the storage device is powered on, and an average amount of data storage in the storage device over a period of time, and
wherein the I/O statistics comprise at least one of: a read rate at a specified period of time, a write rate at the specified period of time, a bit error rate at the specified period of time, and a latency spike in the read rate or write rate.