Methods and apparatus for application prediction through machine learning based analysis of IO patterns
An apparatus comprises a host device configured to communicate over a network with a storage system. The host device comprises a multi-path input-output driver configured to control delivery of input-output operations from the host device to the storage system over selected ones of a plurality of paths through the network. The multi-path input-output driver is further configured to collect information characterizing input-output patterns for each of a plurality of different applications executing on the host device, and to provide the collected information to a machine learning system. The machine learning system utilizes the collected information to generate one or more application predictions based at least in part on analysis of observed input-output patterns. The collected information in some embodiments comprises statistical distributions of numbers of input-output operations having payload sizes falling into specified ranges over a given time period.
1. An apparatus comprising:
a host device configured to communicate over a network with a storage system comprising a plurality of storage devices;
the host device comprising a multi-path input-output driver configured to control delivery of input-output operations from the host device to the storage system over selected ones of a plurality of paths through the network;
wherein the multi-path input-output driver is further configured:
to collect information characterizing input-output patterns for each of a plurality of different applications executing on the host device; and
to provide the collected information to a machine learning system;
wherein the machine learning system utilizes the collected information to generate one or more application predictions based at least in part on analysis of observed input-output patterns;
wherein the machine learning system generates a first application prediction indicating a first application type responsive to detection of a first input-output pattern, and generates a second application prediction indicating a second application type different than the first application type responsive to detection of a second input-output pattern different than the first input-output pattern;
wherein collecting information characterizing input-output patterns for each of a plurality of different applications executing on the host device comprises determining, based at least in part on one or more counts of read requests, write requests or other types of input-output operations, statistical distributions of numbers of input-output operations having payload sizes falling within specified ranges over a given time period; and
wherein the one or more application predictions generated by the machine learning system are utilized to modify a path selection algorithm implemented by the multi-path input-output driver for selecting paths for delivery of input-output operations from the host device to the storage system through the network.
2. The apparatus of claim 1 wherein the machine learning system is implemented at least in part in the host device.
3. The apparatus of claim 2 wherein the machine learning system is implemented at least in part in the multi-path input-output driver of the host device.
4. The apparatus of claim 1 wherein the machine learning system is implemented at least in part in the storage system.
5. The apparatus of claim 1 wherein collecting information characterizing input-output patterns for each of a plurality of different applications executing on the host device comprises collecting information for use in development of a machine learning model for deployment in the machine learning system.
6. The apparatus of claim 5 wherein the collected information for use in development of the machine learning model comprises one or more of:
a training set for training the machine learning model to recognize particular types of applications from their respective input-output patterns; and
a testing set for evaluating performance of the machine learning model in recognizing particular types of applications from their respective input-output patterns.
7. The apparatus of claim 1 wherein collecting information characterizing input-output patterns for each of a plurality of different applications executing on the host device comprises collecting information for use by a deployed machine learning model of the machine learning system in generating the one or more application predictions.
8. The apparatus of claim 1 wherein the multi-path input-output driver is further configured:
to receive the one or more application predictions generated by the machine learning system.
9. The apparatus of claim 1 wherein a given application prediction generated by the machine learning system comprises information identifying a particular one of a plurality of possible application types.
10. The apparatus of claim 1 wherein at least one of the host device and the storage system is configured to utilize the one or more application predictions generated by the machine learning system to adjust one or more parameters related to its processing of input-output operations for the corresponding applications.
11. The apparatus of claim 10 wherein at least one of the host device and the storage system is further configured:
to allocate a relatively high level of processing resources to input-output operations associated with a first application prediction; and
to allocate a relatively low level of processing resources to input-output operations associated with a second application prediction different than the first application prediction.
12. The apparatus of claim 1 further comprising one or more additional host devices each configured to communicate over the network with the storage system and wherein each additional host device comprises a multi-path input-output driver configured to control delivery of input-output operations from that host device to the storage system over selected ones of a plurality of paths through the network.
13. The apparatus of claim 1 wherein the paths are associated with respective initiator-target pairs wherein each of the initiators comprises a corresponding one of a plurality of host bus adaptors of the host device and each of the targets comprises a corresponding one of a plurality of ports of the storage system.
14. A method comprising:
controlling, in a multi-path input-output driver of a host device, delivery of input-output operations from the host device to a storage system over selected ones of a plurality of paths through a network;
collecting, in the multi-path input-output driver, information characterizing input-output patterns for each of a plurality of different applications executing on the host device; and
providing the collected information from the multi-path input-output driver to a machine learning system;
wherein the machine learning system utilizes the collected information to generate one or more application predictions based at least in part on analysis of observed input-output patterns;
wherein the machine learning system generates a first application prediction indicating a first application type responsive to detection of a first input-output pattern, and generates a second application prediction indicating a second application type different than the first application type responsive to detection of a second input-output pattern different than the first input-output pattern;
wherein collecting information characterizing input-output patterns for each of a plurality of different applications executing on the host device comprises determining, based at least in part on one or more counts of read requests, write requests or other types of input-output operations, statistical distributions of numbers of input-output operations having payload sizes falling within specified ranges over a given time period; and
wherein the one or more application predictions generated by the machine learning system are utilized to modify a path selection algorithm implemented by the multi-path input-output driver for selecting paths for delivery of input-output operations from the host device to the storage system through the network.
15. The method of claim 14 wherein the multi-path input-output driver is further configured:
to receive the one or more application predictions generated by the machine learning system.
16. The method of claim 14 wherein at least one of the host device and the storage system utilizes the one or more application predictions generated by the machine learning system to adjust one or more parameters related to its processing of input-output operations for the corresponding applications.
17. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code, when executed by a host device comprising a multi-path input-output driver, the host device being configured to communicate over a network with a storage system, causes the host device:
to control, in the multi-path input-output driver of the host device, delivery of input-output operations from the host device to the storage system over selected ones of a plurality of paths through the network;
to collect, in the multi-path input-output driver, information characterizing input-output patterns for each of a plurality of different applications executing on the host device; and
to provide the collected information from the multi-path input-output driver to a machine learning system;
wherein the machine learning system utilizes the collected information to generate one or more application predictions based at least in part on analysis of observed input-output patterns;
wherein the machine learning system generates a first application prediction indicating a first application type responsive to detection of a first input-output pattern, and generates a second application prediction indicating a second application type different than the first application type responsive to detection of a second input-output pattern different than the first input-output pattern;
wherein collecting information characterizing input-output patterns for each of a plurality of different applications executing on the host device comprises determining, based at least in part on one or more counts of read requests, write requests or other types of input-output operations, statistical distributions of numbers of input-output operations having payload sizes falling within specified ranges over a given time period; and
wherein the one or more application predictions generated by the machine learning system are utilized to modify a path selection algorithm implemented by the multi-path input-output driver for selecting paths for delivery of input-output operations from the host device to the storage system through the network.
18. The computer program product of claim 17 wherein the multi-path input-output driver is further configured:
to receive the one or more application predictions generated by the machine learning system.
19. The computer program product of claim 17 wherein at least one of the host device and the storage system utilizes the one or more application predictions generated by the machine learning system to adjust one or more parameters related to its processing of input-output operations for the corresponding applications.
20. The computer program product of claim 17 wherein a given application prediction generated by the machine learning system comprises information identifying a particular one of a plurality of possible application types.