Systems and methods for deployments of machine learning models for multiple software frameworks, hardware targets, parallelization and distribution modes

1 . A system configured to deploy individual machine learning models for multiple deployment targets, the system comprising:

electronic storage configured to electronically store information; and

one or more hardware processors configured by machine readable instructions to:

obtain a logical representation of a particular machine learning model;

obtain a set of deployment-agnostic parameters for the logical representation of the particular machine learning model;

obtain, from a user, a mode of use for deployments of the particular machine learning model, wherein the mode of use is selected from a set of modes that includes a training mode of use, a fine-tuning mode of use, and an inference mode of use;

obtain, from a user, information regarding multiple deployment targets for the deployments of the particular machine learning model, including a first deployment target for a first deployment of the particular machine learning model and a second deployment target for a second deployment of the particular machine learning model, wherein the information regarding the multiple deployment targets includes one or more of parallelization parameters and/or distribution parameters;

generate a hierarchical directed acyclic graph (DAG) that corresponds to the logical representation of the particular machine learning model, based on the set of deployment-agnostic parameters and the mode of use of the deployments;

perform one or more logical optimizations on the hierarchical DAG to create a modified DAG, wherein the one or more logical optimizations are agnostic with respect to the multiple deployment targets, and wherein the modified DAG is still functionally equivalent to the logical representation of the particular machine learning model;

flatten the modified DAG into a flattened DAG;

perform one or more physical optimizations on the flattened DAG, wherein the one or more physical optimizations are target-specific optimizations based on the information regarding the multiple deployment targets for the deployments of the particular machine learning model, and wherein the flattened DAG after the one or more physical optimizations is still functionally equivalent to the logical representation of the particular machine learning model;

generate a first set of target code that is executable on a first hardware target as the first deployment of the particular machine learning model, based on the flattened DAG as optimized through the one or more logical optimizations and the one or more physical optimizations;

deploy the first deployment of the particular machine learning model using the first set of target code;

generate a second set of target code that is executable on a second hardware target as the second deployment of the particular machine learning model, based on the flattened DAG as optimized through the one or more logical optimizations and the one or more physical optimizations;

deploy the second deployment of the particular machine learning model using the second set of target code, wherein the second deployment is functionally equivalent to the first deployment of the particular machine learning model;

determine a first set of one or more performance metrics associated with execution of the first set of target code for the first deployment of the particular machine learning model, wherein the first set of one or more performance metrics includes a first range of latencies associated with execution of the first set of target code for the first deployment of the particular machine learning model; and

determine a second set of one or more performance metrics associated with execution of the second set of target code for the second deployment of the particular machine learning model, wherein the second set of one or more performance metrics includes a second range of latencies associated with execution of the second set of target code for the second deployment of the particular machine learning model.

2 . The system of claim 1 , wherein the one or more logical optimizations on the modified DAG include one or more of:

adjusting floating point precisions,

pruning one or more computational paths in the modified DAG by virtue of unused inputs and/or unused operations,

simplifying the modified DAG based on one or more ranges of one or more input values, and/or

detecting one or more duplicate computations and reducing the modified DAG by removing the one or more duplicate computations.

3 . The system of claim 1 , wherein the one or more physical optimizations on the flattened DAG further include one or more

universal optimizations that are agnostic with respect to the multiple deployment targets.

4 . The system of claim 1 , wherein the set of deployment-agnostic parameters is obtained through a file provided by the user, and wherein the file is stored in the electronic storage.

5 . The system of claim 1 , wherein the flattened DAG corresponds to a representation of the particular machine learning model that characterizes one or more data types, memory layout, and/or tensor shapes.

6 . The system of claim 1 , wherein generation of the first set of target code is based on one or more of minimizing peak memory usage, maximizing core utilization, and/or optimizing memory performance in accordance with the mode of use.

7 . The system of claim 1 , wherein the logical representation of the particular machine learning model is described in PYTHON™ via a domain-specific language (DSL), and stored in the electronic storage.

8 . The system of claim 1 , wherein the one or more hardware processors are further configured to:

determine one or more data structures that encode one or more dependencies of at least one of the hierarchical DAG, the modified DAG, and/or the flattened DAG, wherein at least one of the one or more logical optimizations and the one or more physical optimizations is based on using the one or more data structures.

9 . The system of claim 1 , wherein the parallelization parameters describe a number of physical computational devices available in at least one of the multiple deployment targets.

10 . The system of claim 9 , wherein optimizing the hierarchical DAG through the one or more logical optimizations and the one or more physical optimizations includes parallelizing computation across the number of physical computational devices based on tensor calculation times and inter-device communication latencies between the number of physical computational devices.

11 . The system of claim 1 , wherein the distribution parameters embody a strategy for sharding tensors across multiple devices and/or nodes.

12 . The system of claim 1 , wherein the one or more hardware processors are further configured to:

determine a first cost associated with the execution of the first set of target code for the first deployment of the particular machine learning model; and

determine a second cost associated with execution of the second set of target code for the second deployment of the particular machine learning model.

13 . The system of claim 1 , wherein the second set of target code includes SystemVerilog code, and wherein the second deployment implements the SystemVerilog code on a field programmable gate array (FPGA) device or platform.

14 . The system of claim 1 , wherein the first deployment target and the second deployment target use different hardware architectures and different instruction set architectures.

15 . The system of claim 1 , wherein the first deployment target uses an architecture based on graphical processing units (GPUs) by NVIDIA™ and the second deployment target uses at least one of a tensor processing unit (TPU) architecture by GOOGLE™, a NeuronCore™ architecture by AWS™, an XDNA™ architecture or an CDNA™ architecture by AMD™, a neural processing unit (NPU) architecture by INTEL™, and/or an AZURE AI architecture by MICROSOFT™.

16 . The system of claim 1 , wherein the multiple deployment targets include a third deployment target for a third deployment of the particular machine learning model, and wherein the one or more hardware processors are further configured to:

generate a third set of target code that is executable as the third deployment of the particular machine learning model, based on the flattened DAG as optimized through the one or more logical optimizations and the one or more physical optimizations;

deploy the third deployment of the particular machine learning model using the third set of target code; and

determine a third set of one or more performance metrics associated with execution of the third set of target code for the third deployment of the particular machine learning model.

17 . The system of claim 1 , wherein the one or more hardware processors are further configured to:

obtain, from the user, a request for execution of a task on the particular machine learning model;

obtain, from the user, a selection for either the first deployment or the second deployment of the particular machine learning model for the execution of the task; and

launch the task for the execution on either the first deployment or the second deployment of the particular machine learning model, in accordance with the selection by the user.

18 . A method of deploying individual machine learning models for multiple deployment targets, the method comprising:

obtaining a logical representation of a particular machine learning model;

obtaining a set of deployment-agnostic parameters for the logical representation of the particular machine learning model;

obtaining, from a user, information regarding multiple deployment targets for deployments of the particular machine learning model, including a first deployment target for a first deployment of the particular machine learning model and a second deployment target for a second deployment of the particular machine learning model, wherein the information regarding the multiple deployment targets includes one or more of parallelization parameters and/or distribution parameters;

generating a hierarchical directed acyclic graph (DAG) that corresponds to the logical representation of the particular machine learning model, based on the set of deployment-agnostic parameters and a mode of use for the deployments of the particular machine learning model;

performing one or more logical optimizations on the hierarchical DAG to create a modified DAG, wherein the one or more logical optimizations are agnostic with respect to the multiple deployment targets, and wherein the modified DAG is still functionally equivalent to the logical representation of the particular machine learning model;

flattening the modified DAG into a flattened DAG;

performing one or more physical optimizations on the flattened DAG, wherein the one or more physical optimizations are target-specific optimizations based on the information regarding the multiple deployment targets for the deployments of the particular machine learning model, and wherein the flattened DAG after the one or more physical optimizations is still functionally equivalent to the logical representation of the particular machine learning model;

generating a first set of target code that is executable on a first hardware target as the first deployment of the particular machine learning model, based on the flattened DAG as optimized through the one or more logical optimizations and the one or more physical optimizations;

deploying the first deployment of the particular machine learning model using the first set of target code;

generating a second set of target code that is executable on a second hardware target as the second deployment of the particular machine learning model, based on the flattened DAG as optimized through the one or more logical optimizations and the one or more physical optimizations;

deploying the second deployment of the particular machine learning model using the second set of target code, wherein the second deployment is functionally equivalent to the first deployment of the particular machine learning model;

determining a first set of one or more performance metrics associated with execution of the first set of target code for the first deployment of the particular machine learning model, wherein the first set of one or more performance metrics includes a first range of latencies associated with execution of the first set of target code for the first deployment of the particular machine learning model; and

determining a second set of one or more performance metrics associated with execution of the second set of target code for the second deployment of the particular machine learning model, wherein the second set of one or more performance metrics includes a second range of latencies associated with execution of the second set of target code for the second deployment of the particular machine learning model.

19 . The method of claim 18 , further comprising:

obtaining, from the user, a request for execution of a task on the particular machine learning model;

obtaining, from the user, a selection for either the first deployment or the second deployment of the particular machine learning model for the execution of the task; and

launching the task for the execution on either the first deployment or the second deployment of the particular machine learning model, in accordance with the selection by the user.