Decoupled architecture for artificial intelligence model management
A decoupled artificial intelligence model management architecture comprises a front-end component, a scheduler component and a back-end component configured to enable an open and dynamic system which accepts any models from any formats to be deployed on any target devices and to dynamically change scheduling of parallelism, parameter movements and computation executions across the target devices.
1. A system, comprising:
a processing platform with at least one processor and at least one memory storing computer program instructions wherein, when the at least one processor executes the computer program instructions, the system further comprises:
a front-end component configured to: receive a selected first one of a set of artificial intelligence models having different initial model formats; translate the selected first artificial intelligence model from an initial model format into a second model format wherein the second model format corresponds to a model format supported by a selected one of a set of target devices upon which the selected first artificial intelligence model is to be executed, wherein the set of target devices are deployed in a computing environment; and generate a computation graph corresponding to the translated artificial intelligence model, wherein the computation graph is generated by compiling the selected first artificial intelligence model to be executed into an intermediate representation, and wherein the computation graph comprises two or more nodes each representing a layer operator defined by the selected first artificial intelligence model and one or more edges defining a dependency relationship between the two or more nodes;
a scheduler component configured to: select a scheduler of a plurality of schedulers based on a selected parallelism algorithm, wherein each of the plurality of schedulers implements a different parallelism algorithm; use the selected scheduler to obtain binary representations of computations based on the two or more nodes of the translated computation graph and dependency rules for the computations based on the one or more edges of the translated computation graph; schedule the computations for execution on the selected one of the target devices based on the dependency rules, one or more characteristics of the computations and the selected parallelism algorithm; and schedule parameters associated with the second model format, the scheduling comprising determining whether the parameters associated with the second model format are registered and sharing the parameters associated with the second model format among a subset of the computations that are scheduled for execution on the selected one of the target devices; and
a back-end component configured to: generate code for executing the scheduled computations based on the second model format of the selected one of the target devices, the generating including generating implementations for the two or more nodes of the computation graph and just-in-time computing the implementations into binary representations of the two or more nodes of the computation graph; and causing the code to be executed on the selected one of the target devices in order to execute at least part of the selected first artificial intelligence model on the selected one of the target devices;
wherein the front-end component, the scheduler component and the back-end component are functionally separate from each other and are further configured to re-perform at least a portion of the above operations corresponding to receipt of a selected second one of the set of artificial intelligence models, or a detected change in at least one of the selected one of the set of target devices and the selected parallelism algorithm.
2. The system of claim 1 , wherein the computing environment in which the set of target devices are deployed comprises an edge computing environment.
3. The system of claim 1 , wherein the selected first artificial intelligence model comprises an inference program.
4. The system of claim 1 , wherein the selected parallelism algorithm comprises a model parallelism algorithm or a pipeline parallelism algorithm.
5. The system of claim 1 , wherein the front-end component is further configured to generate an intermediate representation corresponding to the translated artificial intelligence model.
6. The system of claim 1 , wherein, when the selected one of the set of target devices is a central processing unit, the back-end component is further configured to generate code for execution of all computations on the central processing unit.
7. The system of claim 1 , wherein, when the selected one of the set of target devices is a graphical processing unit, the back-end component is further configured to: generate code for execution on the graphical processing unit for computations that are appropriate for execution on the graphical processing unit; and generate code for execution on a central processing unit for computations that are not appropriate for execution on the graphical processing unit.
8. The system of claim 1 , wherein, when the selected one of the set of target devices is a proprietary target device, the back-end component is further configured to:
generate code for execution of computations on the proprietary target device when available for the proprietary target device; and generate code for execution of computations on a central processing unit when not available for the proprietary target device.
9. The system of claim 1 , wherein the scheduler component is further configured to parse the computation graph.
10. The system of claim 1 , wherein the scheduler component is further configured to bind the parameters associated with the second model format to each computation.
11. The system of claim 10 , wherein a given parameter is shared between two or more computations.
12. A method, comprising:
receiving, in a front-end component, a selected first one of a set of artificial intelligence models having different initial model formats;
translating, in the front-end component, the selected first artificial intelligence model from an initial model format into a second model format wherein the second model format corresponds to a model format supported by a selected one of a set of target devices upon which the selected first artificial intelligence model is to be executed, wherein the set of target devices are deployed in a computing environment;
generating, in the front-end component, a computation graph corresponding to the translated artificial intelligence model, wherein the computation graph is generated by compiling the selected first artificial intelligence model to be executed into an intermediate representation, and wherein the computation graph comprises two or more nodes each representing a layer operator defined by the selected first artificial intelligence model and one or more edges defining a dependency relationship between the two or more nodes;
selecting, with a scheduler component, a scheduler of a plurality of schedulers based on a selected parallelism algorithm, wherein each of the plurality of schedulers implements a different parallelism algorithm;
using the selected scheduler to obtain, in the scheduler component, binary representations of computations based on the two or more nodes of the translated computation graph and dependency rules for the computations based on the one or more edges of the translated computation graph;
scheduling, by the scheduler component, the computations for execution on the selected one of the target devices based on the dependency rules, one or more characteristics of the computations and a selected parallelism algorithm;
scheduling parameters associated with the second model format, the scheduling comprising determining whether the parameters associated with the second model format are registered and sharing the parameters associated with the second model format among a subset of the computations that are scheduled for execution on the selected one of the target devices;
generating, in a back-end component, code for executing the scheduled computations based on the second model format of the selected one of the target devices, the generating including generating implementations for the two or more nodes of the computation graph and just-in-time computing the implementations into binary representations of the two or more nodes of the computation graph;
causing, by the back-end component, the code to be executed on the selected one of the target devices in order to execute at least part of the selected first artificial intelligence model on the selected one of the target devices; and
re-performing at least a portion of the above steps corresponding to receipt of a selected second one of the set of artificial intelligence models or a detected change in at least one of the selected one of the set of target devices and the selected parallelism algorithm;
wherein the front-end component, the scheduler component and the back-end component are functionally separate from each other and are implemented on a processing platform with at least one processor and at least one memory storing computer program instructions wherein, when the at least one processor executes the computer program instructions, the above steps are performed.
13. The method of claim 12 , wherein the computing environment in which the set of target devices are deployed comprises an edge computing environment.
14. The method of claim 12 , wherein the selected first artificial intelligence model comprises an inference program.
15. The method of claim 12 , wherein the selected parallelism algorithm comprises a model parallelism algorithm or a pipeline parallelism algorithm.
16. The method of claim 12 , further comprising generating, in the front-end component, an intermediate representation corresponding to the translated artificial intelligence model.
17. A computer program product stored on a non-transitory computer-readable medium and comprising machine executable instructions, the machine executable instructions, when executed, causing a processing device to:
receive, in a front-end component, a selected first one of a set of artificial intelligence models having different initial model formats;
translate, in the front-end component, the selected first artificial intelligence model from an initial model format into a second model format wherein the second model format corresponds to a model format supported by a selected one of a set of target devices upon which the selected first artificial intelligence model is to be executed, wherein the set of target devices are deployed in a computing environment;
generate, in the front-end component, a computation graph corresponding to the translated artificial intelligence model, wherein the computation graph is generated by compiling the selected first artificial intelligence model to be executed into an intermediate representation, and wherein the computation graph comprises two or more nodes each representing a layer operator defined by the selected first artificial intelligence model and one or more edges defining a dependency relationship between the two or more nodes;
select, with a scheduler component, a scheduler of a plurality of schedulers based on a selected parallelism algorithm, wherein each of the plurality of schedulers implements a different parallelism algorithm;
use the selected scheduler to obtain, in the scheduler component, binary representations of computations based on the two or more nodes of the translated computation graph and dependency rules for the computations based on the one or more edges of the translated computation graph;
schedule, by the scheduler component, the computations for execution on the selected one of the target devices based on the dependency rules and a selected parallelism algorithm;
schedule parameters associated with the second model format, the scheduling comprising determining whether the parameters associated with the second model format are registered and sharing the parameters associated with the second model format among a subset of the computations that are scheduled for execution on the selected one of the target devices;
generate, in a back-end component, code for executing the scheduled computations based on the second model format of the selected one of the target devices, the generating including generating implementations for the two or more nodes of the computation graph and just-in-time computing the implementations into binary representations of the two or more nodes of the computation graph;
cause, by the back-end component, the code to be executed on the selected one of the target devices in order to execute at least part of the selected first artificial intelligence model on the selected one of the target devices; and
re-perform at least a portion of the above steps corresponding to receipt of a selected second one of the set of artificial intelligence models or a detected change in at least one of the selected one of the set of target devices and the selected parallelism algorithm;
wherein the front-end component, the scheduler component and the back-end component are functionally separate from each other.
18. The computer program product of claim 17 , wherein the computing environment in which the set of target devices are deployed comprises an edge computing environment.
19. The computer program product of claim 17 , wherein the selected first artificial intelligence model comprises an inference program.
20. The computer program product of claim 17 , wherein the selected parallelism algorithm comprises a model parallelism algorithm or a pipeline parallelism algorithm.