PIPELINE MONITORING AND RECONFIGURATION IN MACHINE LEARNING SYSTEMS
Execution of a pipeline of a client system and access patterns to one or more storage resources used by the pipeline is monitored. A bottleneck in the pipeline is identified based at least in part on the access patterns. A reconfiguration of resources is initiated, including reallocating compute resources or storage resources, to resolve the bottleneck in the pipeline.
1 . A method comprising:
monitoring execution of a pipeline of a client system and access patterns to one or more storage resources used by the pipeline;
identifying a bottleneck in the pipeline based at least in part on the access patterns; and
initiating a reconfiguration of resources, including reallocating compute resources or storage resources, to resolve the bottleneck in the pipeline.
2 . The method of claim 1 , further comprising:
creating auditing information for the pipeline associated with execution of a machine learning model by the client system.
3 . The method of claim 1 , further comprising:
creating trending information for the pipeline, including performance trends of a machine learning model executed by the client system.
4 . The method of claim 1 , further comprising:
detecting data drift with a machine learning model executed by the client system.
5 . The method of claim 1 , further comprising:
detecting, based on the monitoring, a change in data distribution within training data processed in the pipeline.
6 . The method of claim 1 , wherein initiating the reconfiguration comprises:
reallocating compute resources of the client system to stages of the pipeline.
7 . The method of claim 1 , wherein initiating the reconfiguration comprises:
reallocating storage resources of the client system to stages of the pipeline.
8 . The method of claim 1 , wherein the monitoring comprises evaluating log files generated by the client system during execution of the pipeline to identify an execution pattern associated with a machine learning model.
9 . The method of claim 8 , further comprising:
comparing the execution pattern to a known execution pattern to identify the bottleneck in execution of the machine learning model.
10 . The method of claim 1 , wherein initiating the reconfiguration of resources comprises:
storing data associated with the pipeline in cache memory.
11 . An apparatus comprising:
a memory; and
a processing device, operatively coupled to the memory, configured to:
monitor execution of a pipeline of a client system and access patterns to one or more storage resources used by the pipeline;
identify a bottleneck in the pipeline based at least in part on the access patterns; and
initiate a reconfiguration of resources, including reallocating compute resources or storage resources, to resolve the bottleneck in the pipeline.
12 . The apparatus of claim 11 , wherein the processing device is further configured to:
create auditing information for the pipeline associated with execution of a machine learning model by the client system.
13 . The apparatus of claim 11 , wherein the processing device is further configured to:
create trending information for the pipeline, including performance trends of a machine learning model executed by the client system.
14 . The apparatus of claim 11 , wherein the processing device is further configured to:
detect data drift with a machine learning model executed by the client system.
15 . The apparatus of claim 11 , wherein the processing device is further configured to:
detect, based on the monitoring, a change in data distribution within training data processed in the pipeline.
16 . The apparatus of claim 11 , wherein to initiate the reconfiguration, the processing device is further configured to:
reallocate compute resources of the client system to stages of the pipeline.
17 . The apparatus of claim 11 , wherein to initiate the reconfiguration, the processing device is further configured to:
reallocate storage resources of the client system to stages of the pipeline.
18 . The apparatus of claim 11 , wherein the monitoring comprises evaluating log files generated by the client system during execution of the pipeline to identify an execution pattern associated with a machine learning model.
19 . The apparatus of claim 18 , wherein the processing device is further configured to:
compare the execution pattern to a known execution pattern to identify the bottleneck in execution of the machine learning model.
20 . A non-transitory computer readable storage medium storing instructions which, when executed, cause a processing device to:
monitor execution of a pipeline of a client system and access patterns to one or more storage resources used by the pipeline;
identify a bottleneck in the pipeline based at least in part on the access patterns; and
initiate a reconfiguration of resources, including reallocating compute resources or storage resources, to resolve the bottleneck in the pipeline.