Machine learning model replacement on edge devices
Techniques for machine learning prediction using edge devices are described. In some examples a method of use includes receiving a request to load a second model onto an edge device while a first model has already been loaded on the edge device, wherein the second model and the first model share an external handle; loading at least one instance the second model into memory of the edge device; and after the second model has been loaded into memory of the edge device, directing a prediction request to the shared external handle to the second model instead of the first model.
1. A computer-implemented method comprising:
receiving a request to load a second machine learning (ML) model onto an edge device while a first ML model has already been loaded into memory of the edge device, wherein the second ML model and the first ML model share an external handle but have different internal aliases;
in response to the request, the edge device executing edge compute code using one or more processors to load at least one instance of the second ML model into the memory of the edge device while the first ML model is still loaded into the memory of the edge device;
after the second ML model has been loaded into the memory of the edge device, receiving a prediction request at the shared external handle of the first ML model and the second ML model;
in response to the prediction request, the edge device executing the edge compute code using the one or more processors to perform a handle-to-alias translation to determine to direct the prediction request to the second ML model instead of the first ML model;
directing the prediction request to the second ML model instead of the first ML model; and
after directing the prediction request to the second ML model instead of the first ML model, unloading the first ML model from the memory of the edge device.
2. The computer-implemented method of claim 1 , wherein the request to load the second ML model onto the edge device while the first ML model has already been loaded into the memory of the edge device includes at least one of an identification of a model handle, an identification of a model alias, an indication of a location of a model and/or its runtime, an indication of input and/or output buffer sizes, or an indication of a desired execution environment.
3. The computer-implemented method of claim 1 , wherein the second ML model is a variant of the first ML model.
4. A computer-implemented method comprising:
receiving a request to load a second machine learning (ML) model onto an edge device while a first ML model has already been loaded into memory of the edge device, wherein the second ML model and the first ML model share an external handle but have different internal aliases;
in response to the request, the edge device executing edge compute code using one or more processors to load at least one instance of the second ML model into the memory of the edge device while the first ML model is still loaded into the memory of the edge device;
after the second ML model has been loaded into the memory of the edge device, receiving a prediction request at the shared external handle of the first ML model and the second ML model;
in response to the prediction request, the edge device executing the edge compute code using the one or more processors to perform a handle-to-alias translation to determine to direct the prediction request to the second ML model instead of the first ML model; and
directing the prediction request to the second ML model instead of the first ML model.
5. The computer-implemented method of claim 4 , wherein the second ML model is a variant of the first ML model.
6. The computer-implemented method of claim 4 , wherein routing is based on the aliases.
7. The computer-implemented method of claim 4 , wherein loading the at least one instance of the second ML model into the memory of the edge device at least includes loading a runtime.
8. The computer-implemented method of claim 4 , further comprising batching requests to the shared external handle, wherein directing the prediction request to the second ML model instead of the first ML model is a part of a batch.
9. The computer-implemented method of claim 4 , further comprising unloading the first ML model from the memory of the edge device.
10. The computer-implemented method of claim 9 , wherein the first ML model is unloaded in response to an unload request.
11. The computer-implemented method of claim 10 , wherein the unload request includes an identification of the shared external handle and a model alias.
12. The computer-implemented method of claim 9 , wherein the first ML model is unloaded to free the memory into which the first ML model had been loaded.
13. The computer-implemented method of claim 4 , wherein the request to load the second ML model onto the edge device while the first ML model has already been loaded into the memory of the edge device includes at least one of an identification of a model handle, an identification of a model alias, an indication of a location of a model and/or its runtime, an indication of input and/or output buffer sizes, or an indication of a desired execution environment.
14. The computer-implemented method of claim 4 , wherein loading the second ML model comprises downloading and verifying the second ML model.
15. A system comprising:
a first one or more electronic devices to implement an edge device service in a multi-tenant provider network; and
a second one or more electronic devices to implement edge compute software in the multi-tenant provider network, the edge compute software including instructions that upon execution by one or more processors cause the edge compute software to:
receive a request to load a second machine learning (ML) model onto the edge device while a first ML model has already been loaded into memory of the edge device, wherein the second ML model and the first ML model share an external handle but have different internal aliases;
in response to the request, the second one or more electronic devices executing the edge compute software using the one or more processors to load at least one instance of the second ML model into the memory of the edge device while the first ML model is still loaded into the memory of the edge device;
after the second ML model has been loaded into the memory of the edge device, receiving a prediction request at the shared external handle of the first ML model and the second ML model;
in response to the prediction request, the edge device executing the edge compute software using the one or more processors to perform a handle-to-alias translation to determine to direct the prediction request to the second ML model instead of the first ML model; and
direct the prediction request to the second ML model instead of the first ML model.
16. The system of claim 15 , wherein the second ML model is a variant of the first ML model.
17. The system of claim 15 , wherein routing is based on the aliases.
18. The system of claim 15 , wherein the first ML model is to be unloaded in response to an unload request.
19. The system of claim 18 , wherein the unload request includes an identification of the shared external handle and a model alias.
20. The system of claim 15 , wherein the request to load the second ML model onto the edge device while the first ML model has already been loaded into the memory of the edge device includes at least one of an identification of a model handle, an identification of a model alias, an indication of a location of a model and/or its runtime, an indication of input and/or output buffer sizes, or an indication of a desired execution environment.