Machine learning based resource availability prediction
Requests from file system services of a storage system are registered. Each file system service, when executed, utilizes one or more resources of the storage system. Each request includes information describing resource requirements required by a respective file system service. Resource utilization data of the resources are collected over a period of time. The resource utilization data includes an identification of a resource, a timestamp, and a measurement indicating a utilization level of the resource corresponding to the timestamp. A machine learning model is trained to predict utilization patterns of the resources. Execution of the file system services are scheduled based on the predicted utilization patterns. Monitoring is conducted during the execution of the file system services. Based on the monitoring a determination is made as to whether the machine learning model should be retrained.
1. A method of efficiently managing a plurality of resources of a protection storage system comprising:
registering a plurality of requests from a plurality of file system services of the storage system, each file system service when executed utilizing one or more resources, and each request comprising information describing resource requirements required by a respective file system service;
collecting, over a period of time, resource utilization data of the plurality of resources, the resource utilization data comprising an identification of a resource, a timestamp, and a measurement indicating a utilization level of the resource corresponding to the timestamp;
training, using the resource utilization data, a machine learning model to predict utilization patterns of the plurality of resources;
based on the predicted utilization patterns, scheduling execution of the file system services;
conducting monitoring during execution of the file system services; and
based on the monitoring, determining whether the machine learning model should be retrained, wherein the resource utilization data of a resource comprises timestamps, and measurements indicating utilization levels of the resource at times corresponding to the timestamps, and the training the machine learning model further comprises:
processing the resource utilization data for the machine learning model by converting the timestamps to epoch times and generating tuples comprising a first value corresponding to a measurement, and a second value corresponding to an epoch time.
2. The method of claim 1 further comprising:
computing a difference between the predicted utilization patterns and actual utilization patterns;
comparing the difference against an acceptable threshold error; and
when the difference is greater than the acceptable threshold error, retraining the machine learning model.
3. The method of claim 1 further comprising:
monitoring a performance of the protection storage system;
comparing the performance to a threshold performance; and
when the performance falls below the threshold performance, retraining the model.
4. The method of claim 1 wherein a utilization pattern of a resource comprises an availability window indicating starting and ending times at which utilization of the resource is predicted to be low and the method further comprises:
scheduling a file system service to be executed during the availability window, wherein registration information for the file system service indicates a requirement for the resource.
5. The method of claim 1 wherein the information describing resource requirements required by the respective file system resource comprises an expected utilization level of the resource.
6. The method of claim 1 wherein the machine learning model comprises a Long Short Term Memory (LSTM) network.
7. A system comprising: a processor; and memory configured to store one or more sequences of instructions which, when executed by the processor, cause the processor to carry out the steps of:
registering a plurality of requests from a plurality of file system services of a protection storage system, each file system service when executed utilizing one or more resources of a plurality of resources, and each request comprising information describing resource requirements required by a respective file system service;
collecting, over a period of time, resource utilization data of the plurality of resources, the resource utilization data comprising an identification of a resource, a timestamp, and a measurement indicating a utilization level of the resource corresponding to the timestamp;
training, using the resource utilization data, a machine learning model to predict utilization patterns of the plurality of resources;
based on the predicted utilization patterns, scheduling execution of the file system services;
conducting monitoring during execution of the file system services; and
based on the monitoring, determining whether the machine learning model should be retrained, wherein the resource utilization data of a resource comprises timestamps, and measurements indicating utilization levels of the resource at times corresponding to the timestamps, and the training the machine learning model further comprises:
processing the resource utilization data for the machine learning model by converting the timestamps to epoch times and generating tuples comprising a first value corresponding to a measurement, and a second value corresponding to an epoch time.
8. The system of claim 7 wherein the processor further carries out the steps of:
computing a difference between the predicted utilization patterns and actual utilization patterns;
comparing the difference against an acceptable threshold error; and
when the difference is greater than the acceptable threshold error, retraining the machine learning model.
9. The system of claim 7 wherein the processor further carries out the steps of:
monitoring a performance of the protection storage system;
comparing the performance to a threshold performance; and
when the performance falls below the threshold performance, retraining the model.
10. The system of claim 7 wherein a utilization pattern of a resource comprises an availability window indicating starting and ending times at which utilization of the resource is predicted to be low and the processor further carries out the steps of:
scheduling a file system service to be executed during the availability window, wherein registration information for the file system service indicates a requirement for the resource.
11. The system of claim 7 wherein the information describing resource requirements required by the respective file system resource comprises an expected utilization level of the resource.
12. The system of claim 7 wherein the machine learning model comprises a Long Short Term Memory (LSTM) network.
13. A computer program product, comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein, the computer-readable program code adapted to be executed by one or more processors to implement a method comprising:
registering a plurality of requests from a plurality of file system services of a protection storage system, each file system service when executed utilizing one or more resources of a plurality of resources, and each request comprising information describing resource requirements required by a respective file system service;
collecting, over a period of time, resource utilization data of the plurality of resources, the resource utilization data comprising an identification of a resource, a timestamp, and a measurement indicating a utilization level of the resource corresponding to the timestamp;
training, using the resource utilization data, a machine learning model to predict utilization patterns of the plurality of resources;
based on the predicted utilization patterns, scheduling execution of the file system services;
conducting monitoring during execution of the file system services; and
based on the monitoring, determining whether the machine learning model should be retrained, wherein the resource utilization data of a resource comprises timestamps, and measurements indicating utilization levels of the resource at times corresponding to the timestamps, and the training the machine learning model further comprises:
processing the resource utilization data for the machine learning model by converting the timestamps to epoch times and generating tuples comprising a first value corresponding to a measurement, and a second value corresponding to an epoch time.
14. The computer program product of claim 13 wherein the method further comprises:
computing a difference between the predicted utilization patterns and actual utilization patterns;
comparing the difference against an acceptable threshold error; and
when the difference is greater than the acceptable threshold error, retraining the machine learning model.
15. The computer program product of claim 13 wherein the method further comprises:
monitoring a performance of the protection storage system;
comparing the performance to a threshold performance; and
when the performance falls below the threshold performance, retraining the model.
16. The computer program product of claim 13 wherein a utilization pattern of a resource comprises an availability window indicating starting and ending times at which utilization of the resource is predicted to be low and the method further comprises:
scheduling a file system service to be executed during the availability window, wherein registration information for the file system service indicates a requirement for the resource.
17. The computer program product of claim 13 wherein the machine learning model comprises a Long Short Term Memory (LSTM) network.