Distributed artificial intelligence runtime at the network edge as a service
This disclosure describes techniques and mechanisms for enabling a user and third party applications to dynamically partition and place heavy deep learning workloads on standard edge networks to optimize the overall inference throughput of the network while meeting Service Level Objective(s) (SLOs). The techniques may include profiling, partitioning, and splitting of the deep learning workloads, which may be hidden from the user and/or third party application. The user may user interact with a pre-deployed service through a simple SDK that resembles those used for hardware acceleration, such that the current techniques may be easily inserted into their code.
1 . A method implemented at least in part by a controller of a network, the method comprising:
receiving, from an application, a request from a user device to host an artificial intelligence model that is configured to provide a service for users;
determining one or more locations in the artificial intelligence model at which to split the artificial intelligence model;
identifying one or more computing devices in the network at which to run workloads associated with one or more portions of the artificial intelligence model, the one or more computing devices including a first computing device and a second computing device;
identifying a bandwidth metric associated with a network connection between the first computing device and the second computing device;
determining, using the bandwidth metric associated with the network connection between the first computing device and the second computing device, one or more optimal locations at which to split the artificial intelligence model;
splitting the artificial intelligence model at the one or more optimal locations;
deploying a first workload associated with a first portion of the artificial intelligence model to the first computing device;
deploying a second workload associated with a second portion of the artificial intelligence model to the second computing device;
receiving, at the controller and from a user device, a command for the artificial intelligence model to perform a function of the service on behalf of the user device;
causing, by the controller, the artificial intelligence model to perform the function by communicating with the first workload on the first computing device and the second workload on the second computing device, wherein the communicating by the controller is hidden from the user device; and
providing, from the controller and to the user device, a result of the function performed on behalf of the user device.
2 . The method of claim 1 , further comprising:
exposing, by the controller, an interface via a software development kit (SDK), wherein:
the request is received from the user device via the SDK; and
providing the result is performed by sending, via the SDK, the result of the function performed on behalf of the user device.
3 . The method of claim 1 , wherein the request is received from a user computing device associated with a user of the network.
4 . The method of claim 1 , wherein determining the one or more optimal locations includes identifying a split location between one or more layers of the artificial intelligence model, the split location being associated with optimizing throughput of the network.
5 . The method of claim 1 , wherein identifying the one or more computing devices is based at least in part on one or more service level objectives associated with the one or more computing devices.
6 . The method of claim 1 , wherein the request further comprises one or more input or output streams of data.
7 . The method of claim 1 , further comprising presenting, by the controller and to the application, the artificial intelligence model deployment as a virtual hardware acceleration service, wherein the application comprises a software development kit (SDK) configured to enable a third-party developer to interact with the controller using an interface that resembles an interface used for hardware acceleration, and wherein the splitting of the artificial intelligence model and the deploying of the first workload and the second workload by the controller are hidden from the third-party developer.
8 . The method of claim 1 , further comprising:
monitoring, by the controller, one or more network conditions associated with the network, the one or more network conditions including at least one of a CPU load of the first computing device or the second computing device, a bandwidth change associated with the network connection between the first computing device and the second computing device, or a failure of the first computing device or the second computing device;
determining, based on the monitored one or more network conditions, that a recomputation of one or more optimal locations at which to split the artificial intelligence model is warranted; and
in response to determining that the recomputation is warranted:
recomputing, using an updated bandwidth metric associated with the network connection between the first computing device and the second computing device, one or more updated optimal locations at which to split the artificial intelligence model;
splitting the artificial intelligence model at the one or more updated optimal locations; and
redeploying a first updated workload associated with a first updated portion of the artificial intelligence model to the first computing device and a second updated workload associated with a second updated portion of the artificial intelligence model to the second computing device.
9 . A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving, by a controller of a network and from an application, a request from a user device to host an artificial intelligence model that is configured to provide a service for users;
determining, by the controller, one or more locations in the artificial intelligence model at which to split the artificial intelligence model;
identifying, by the controller, one or more computing devices in the network at which to run workloads associated with one or more portions of the artificial intelligence model, the one or more computing devices including a first computing device and a second computing device;
identifying a bandwidth metric associated with a network connection between the first computing device and the second computing device;
determining, using the bandwidth metric associated with the network connection between the first computing device and the second computing device, one or more optimal locations at which to split the artificial intelligence model;
splitting, by the controller, the artificial intelligence model at the one or more optimal locations;
deploying, by the controller, a first workload associated with a first portion of the artificial intelligence model to the first computing device;
deploying, by the controller, a second workload associated with a second portion of the artificial intelligence model to the second computing device;
receiving, at the controller and from a user device, a command for the artificial intelligence model to perform a function of the service on behalf of the user device;
causing, by the controller, the artificial intelligence model to perform the function by communicating with the first workload on the first computing device and the second workload on the second computing device, wherein the communicating by the controller is hidden from the user device; and
providing, from the controller and to the user device, a result of the function performed on behalf of the user device.
10 . The system of claim 9 , the operations further comprising:
exposing, by the controller, an interface via a software development kit (SDK), wherein:
the request is received from the user device via the SDK; and
providing the result is performed by sending, via the SDK, the result of the function performed on behalf of the user device.
11 . The system of claim 9 , wherein the request is received from a user computing device associated with a user of the network.
12 . The system of claim 9 , wherein determining the one or more optimal locations includes identifying a split location between one or more layers of the artificial intelligence model, the split location being associated with optimizing throughput of the network.
13 . The system of claim 9 , wherein identifying the one or more computing devices is based at least in part on one or more service level objectives associated with the one or more computing devices.
14 . The system of claim 9 , wherein the request further comprises one or more input or output streams of data.
15 . One or more non-transitory computer-readable media storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving, from an application, input including an artificial intelligence model;
determining one or more locations in the artificial intelligence model at which to split the artificial intelligence model;
identifying one or more computing devices in a network at which to run workloads associated with one or more portions of the artificial intelligence model, the one or more computing devices including a first computing device and a second computing device;
identifying a bandwidth metric associated with a network connection between the first computing device and the second computing device;
determining, using the bandwidth metric associated with the network connection between the first computing device and the second computing device, one or more optimal locations at which to split the artificial intelligence model;
splitting the artificial intelligence model at the one or more optimal locations;
deploying a first workload associated with a first portion of the artificial intelligence model to the first computing device; and
deploying a second workload associated with a second portion of the artificial intelligence model to the second computing device,
wherein determining the one or more optimal locations is based at least in part on, vector sizes between layers of the artificial intelligence model, or node capacities of the one or more computing devices to optimize inference throughput.
16 . The one or more non-transitory computer-readable media of claim 15 , wherein the application comprises a software development kit (SDK).
17 . The one or more non-transitory computer-readable media of claim 15 , wherein the artificial intelligence model comprises a deep learning neural network.
18 . The one or more non-transitory computer-readable media of claim 15 , wherein determining the one or more optimal locations includes identifying a split location between one or more layers of the artificial intelligence model, the split location being associated with optimizing throughput of the network.
19 . The one or more non-transitory computer-readable media of claim 15 , wherein identifying the one or more computing devices is based at least in part on one or more service level objectives associated with the one or more computing devices.
20 . The one or more non-transitory computer-readable media of claim 15 , wherein the input further comprises one or more input or output streams of data.