Machine learning inference service disaggregation
Aspects of the disclosure are directed to performing disaggregation-aware model graph partitioning, which can include provisioning and load balancing disaggregated resource pools, such as general purpose processors, accelerators, general purpose memory, and high bandwidth memory. Across these disaggregated resource pools, machine learning model operations can be packed and/or batched. The partitioning can further include automatically tuning runtime parameters.
1 . A method for performing machine learning model disaggregation, the method comprising:
generating, with one or more processors, a profile for a machine learning model by running a sample of the machine learning model on a sample dataset of a user project associated with an application of the machine learning model, the profile comprising a resource usage and application type for the machine learning model;
determining, with the one or more processors, a resource threshold to perform the application based on the resource usage and application type of the profile;
determining, with the one or more processors, a graph partition for components of the machine learning model based on the application type of the profile, a data transfer threshold, and connectivity topologies;
validating, with the one or more processors, that resources are available for the graph partition for the components of the machine learning model based on the resource threshold;
partitioning, with the one or more processors, the components of the machine learning model into a plurality of host nodes and at least one accelerator node based on the validating; and
performing the application by executing the machine learning model based on the partitioning.
2 . The method of claim 1 , wherein the host nodes each comprise a processor for parsing operations and a memory comprising embedding tables.
3 . The method of claim 1 , wherein the accelerator node comprises a machine learning accelerator for neural network operations and a high bandwidth memory comprising model parameters or embedding tables.
4 . The method of claim 1 , wherein the data transfer threshold comprises at least one of a network bandwidth, latency, or throughput threshold to reduce hops between the host nodes and the accelerator node.
5 . The method of claim 1 , wherein the connectivity topologies comprise at least one of slicing of accelerators or hierarchies of network topologies.
6 . The method of claim 1 , wherein determining the graph partition for components of the machine learning model is further based on a statistical distribution of the machine learning model.
7 . A system comprising:
one or more processors; and
one or more storage devices coupled to the one or more processors and storing instructions that, when executed by the one or more processors, causes the one or more processors to perform operations for machine learning model disaggregation, the operations comprising:
generating a profile for a machine learning model by running a sample of the machine learning model on a sample dataset of a user project associated with an application of the machine learning model, the profile comprising a resource usage and application type for the machine learning model;
determining a resource threshold to perform the application based on the resource usage and application type of the profile;
determining a graph partition for components of the machine learning model based on the application type of the profile, a data transfer threshold, and connectivity topologies;
validating that resources are available for the graph partition for the components of the machine learning model based on the resource threshold;
partitioning the components of the machine learning model into a plurality of host nodes and at least one accelerator node based on the validating; and
performing the application by executing the machine learning model based on the partitioning.
8 . The system of claim 7 , wherein the host nodes each comprise a processor for parsing operations and a memory comprising embedding tables.
9 . The system of claim 7 , wherein the accelerator node comprises a machine learning accelerator for neural network operations and a high bandwidth memory comprising model parameters or embedding tables.
10 . The system of claim 7 , wherein the data transfer threshold comprises at least one of a network bandwidth, latency, or throughput threshold to reduce hops between the host nodes and the accelerator node.
11 . The system of claim 7 , wherein the connectivity topologies comprise at least one of slicing of accelerators or hierarchies of network topologies.
12 . The system of claim 7 , wherein determining the graph partition for components of the machine learning model is further based on a statistical distribution of the machine learning model.
13 . A non-transitory computer readable medium for storing instructions that, when executed by one or more processors, causes the one or more processors to perform operations for machine learning model disaggregation, the operations comprising:
generating a profile for a machine learning model by running a sample of the machine learning model on a sample dataset of a user project associated with an application of the machine learning model, the profile comprising a resource usage and application type for the machine learning model;
determining a resource threshold to perform the application based on the resource usage and application type of the profile;
determining a graph partition for components of the machine learning model based on the application type of the profile, a data transfer threshold, and connectivity topologies;
validating that resources are available for the graph partition for the components of the machine learning model based on the resource threshold;
partitioning the components of the machine learning model into a plurality of host nodes and at least one accelerator node based on the validating; and
performing the application by executing the machine learning model based on the partitioning.
14 . The non-transitory computer readable medium of claim 13 , wherein the host nodes each comprise a processor for parsing operations and a memory comprising embedding tables.
15 . The non-transitory computer readable medium of claim 13 , wherein the accelerator node comprises a machine learning accelerator for neural network operations and a high bandwidth memory comprising model parameters or embedding tables.
16 . The non-transitory computer readable medium of claim 13 , wherein the data transfer threshold comprises at least one of a network bandwidth, latency, or throughput threshold to reduce hops between the host nodes and the accelerator node.
17 . The non-transitory computer readable medium of claim 13 , wherein the connectivity topologies comprise at least one of slicing of accelerators or hierarchies of network topologies.
18 . The non-transitory computer readable medium of claim 13 , wherein determining the graph partition for components of the machine learning model is further based on a statistical distribution of the machine learning model.