System and method for model orchestration
A system for large-scale machine learning experiment execution, including: a platform configured to determine an experiment set from a run specification and schedule a run to one or more clusters; and a set of agents configured to receive the experiment set from the platform and facilitate individual experiment execution through a cluster orchestrator.
1 . A method comprising:
receiving cluster telemetry, comprising metrics for a set of processes running on a cluster, from an agent for the cluster;
sending control instructions, in a platform-standard protocol determined based on the cluster telemetry, to the agent, wherein the agent converts the platform-standard protocol into a cluster orchestrator-specific protocol and controls a cluster orchestrator associated with the cluster using the cluster orchestrator-specific protocol;
facilitating storage of outputs, generated by the set of processes, within an object store shared across different clusters, wherein a set of processing systems does not have access to the outputs stored in the object store.
2 . The method of claim 1 , further comprising scheduling the set of processes to different machines hosted by different machine providers.
3 . The method of claim 1 , further comprising reconciling the set of processes across different clusters by replacing a stored run state with an actual run state read from the cluster.
4 . The method of claim 1 , further comprising:
accessing a machine provider account for a user using access credentials provided by the user; and
initializing the cluster on behalf of the user using the access credentials.
5 . The method of claim 1 , wherein the cluster orchestrator is configured to control containers executing within the cluster.
6 . The method of claim 1 , wherein the control instructions are determined based on a state machine.
7 . A system, comprising a set of processing systems configured to:
determine a set of runs, each run comprising a set of processes;
schedule the set of runs to clusters, wherein an agent associated with a cluster controls run execution on the cluster; and
concurrently control execution of different sets of runs on different sets of clusters for different users.
8 . The system of claim 7 , wherein the execution of the different sets of runs is indirectly controlled, wherein control instructions are sent to the agent that controls a cluster orchestrator.
9 . The system of claim 7 , wherein the set of processes for a run are executed on a heterogenous computing architecture.
10 . The system of claim 7 , further comprising facilitating storage of outputs, generated by the set of processes, within an object store shared across different clusters, wherein the set of processing systems does not have access to the outputs stored in the object store.
11 . The system of claim 10 , wherein the object store is stored on a machine provided by a cloud provider, a user has access to the outputs through a cloud account for the cloud provider.
12 . The system of claim 7 , wherein the processing systems are further configured to provision each cluster with the agent and a cluster orchestrator.
13 . The system of claim 7 , wherein a run of the set of runs generates a set of trained models.
14 . The system of claim 13 , wherein each trained model can be accessed by a user and be inaccessible to the set of processing systems.
15 . The system of claim 7 , wherein a summary of the set of processes can be visualized by a user, wherein the summary comprises at least one of a run time, a billed time, or a process state.
16 . The system of claim 7 , wherein the set of processing systems is further configured to reconcile the set of processes for a run of the set of runs across different clusters by replacing a stored run state for each run with an actual run state read from the cluster.
17 . A method comprising:
receiving cluster telemetry, comprising metrics for a set of processes running on a cluster, from an agent for a cluster;
sending control instructions based on the cluster telemetry to the agent, wherein the agent controls a cluster orchestrator associated with the cluster based on the control instructions; and
reconciling the set of processes across different clusters by replacing a stored run state with an actual run state read from the cluster.
18 . The method of claim 17 , wherein the agent converts the control instructions to a cluster orchestrator-specific protocol and controls the cluster orchestrator associated with the cluster using the cluster orchestrator-specific protocol.
19 . The method of claim 17 , wherein each process of the set generates a trained model, the method further comprising storing each trained model in a datastore shared across the different clusters.