Distributed application development platform
In variants, the AI/ML development system can include one or more applications, wherein each application can include: one or more components and one or more state storages. Each application can optionally include one or more event loops, one or more shared storages, and/or one or more time schedulers.
1. A system, comprising:
a computing system configured to manage a set of components and a shared state storage system, the set of components comprising a set of flow components and a set of work components, wherein each flow component of the set of flow components defines coordination logic between other components of the set of components, and wherein each work component of the set of work components defines computation logic.
2. The system of claim 1 , wherein the shared state storage system is configured to track a set of component states for the set of components.
3. The system of claim 2 , wherein each component state comprises values for a set of attributes defined within a component.
4. The system of claim 1 , wherein a work component of the set of work components defines computation logic for optimizing a set of parameters, wherein a component state of the set of component states comprises values for the set of parameters.
5. The system of claim 4 , wherein the set of parameters comprises a set of hyperparameters.
6. The system of claim 2 , wherein a flow component of the set of flow components is executed in response to the shared state storage reflecting a change in a component state associated with a work component of the set of work components.
7. The system of claim 1 , wherein the computing system comprises a distributed computing orchestration system.
8. The system of claim 1 , further comprising a shared storage configured to exchange data between different components of the set of components.
9. The system of claim 1 , wherein a work component of the set of work components defines computation logic for training a machine learning model.
10. The system of claim 1 , wherein a work component of the set of work components defines computation logic for running a third-party service.
11. The system of claim 1 , wherein each work component of the set of work components is executed on a different machine.
12. A method, comprising:
initializing a set of components comprising a set of flow components and a set of work components, wherein each flow component of the set of flow components is configured to coordinate execution of other components of the set of components, and wherein each work component of the set of work components is configured to perform a task; and
tracking a set of component states for the set of components.
13. The method of claim 12 , wherein each component of the set of components writes a component state to a shared state storage.
14. The method of claim 12 , wherein a flow component of the set of flow components is executed in response to a change in a component state associated with a work component of the set of work components.
15. The method of claim 12 , wherein the task comprises at least one of:
training a model, optimizing hyperparameters, managing data, or running a third-party service.
16. The method of claim 12 , further comprising triggering execution of the set of components.
17. The method of claim 16 , wherein triggering execution of the set of components comprises triggering a call to a root flow component of the set of flow components, wherein the root flow component calls a subset of the set of components referenced by the root flow component.
18. The method of claim 16 , wherein the set of components are executed on at least one of a developer infrastructure, a platform infrastructure, or a user infrastructure.
19. The method of claim 16 , wherein each work component of the set of work components is executed on a different machine.
20. The method of claim 12 , wherein each component state comprises values for a set of attributes defined within a component.