Providing application programming interface endpoints for machine learning models
One or more virtual machines are launched at an application platform. At each of the one or more virtual machines, a machine learning model execution environment is instantiated for an instance of a machine learning model. A respective instance of the machine learning model is loaded to each machine learning model execution environment. Each loaded instance of the machine learning model is associated with an application programming interface (API) endpoint which can receive input data for the loaded instance of the machine learning model from a client device and return output data produced by the loaded instance of the machine learning model based on the input data.
1. A method, comprising:
receiving a request, from a client device, the request comprising input data for a desired machine learning model;
selecting an execution environment with a loaded instance of the desired machine learning model by:
identifying one or more active virtual machines that are associated with an application programming interface (API) endpoint;
determining a virtual machine of the one or more active virtual machines with an available execution environment that has the loaded instance of the desired machine learning model; and
selecting the available execution environment to be the execution environment; and
instructing the API endpoint to forward the input data to the selected execution environment to obtain output data that is returned to the client device.
2. The method of claim 1 , wherein the API endpoint is associated with the desired machine learning model, and wherein the request includes an identifier of the API endpoint.
3. The method of claim 1 , wherein the determined virtual machine is not being used in conjunction with input data of any other client device.
4. The method of claim 1 , further comprising:
preloading a dataset associated with the desired machine learning model into one or more memories that is accessible by the selected execution environment.
5. The method of claim 1 , further comprising:
generating a notification indicating the selected execution environment is ready for operation, the notification including an address of the API endpoint.
6. A system, comprising:
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, cause the system to perform a set of operations, the set of operations comprising:
receiving a request, from a client device, the request comprising input data for a desired machine learning model;
selecting an execution environment with a loaded instance of the desired machine learning model by:
identifying one or more active virtual machines that are associated with an application programming interface (API) endpoint;
determining a virtual machine of the one or more active virtual machines with an available execution environment that has the loaded instance of the desired machine learning model; and
selecting the available execution environment to be the execution environment; and
instructing the API endpoint to forward the input data to the selected execution environment to obtain output data that is returned to the client device.
7. The system of claim 6 , wherein the API endpoint is associated with the desired machine learning model, and wherein the request includes an identifier of the API endpoint.
8. The system of claim 6 , wherein the determined virtual machine is not being used in conjunction with input data of any other client device.
9. The system of claim 6 , wherein the input data comprises client identifiers that are associated with client account information, and wherein the client account information comprises one or more from the group of: client location, gender, account details, and previous purchases.
10. The system of claim 6 , wherein the set of operations further comprises:
preloading a dataset associated with the desired machine learning model into one or more memories that is accessible by the selected execution environment.
11. The system of claim 6 , wherein the set of operations further comprises:
generating a notification indicating the selected execution environment is ready for operation, the notification including an address of the API endpoint.
12. A non-transitory computer-readable storage medium comprising instructions that when executed by one or more processors cause the one or more processors to perform a set of operations comprising:
receiving a request, from a client device, the request comprising input data for a desired machine learning model;
selecting an execution environment with a loaded instance of the desired machine learning model by:
identifying one or more active virtual machines that are associated with an application programming interface (API) endpoint;
determining a virtual machine of the one or more active virtual machines with an available execution environment that has the loaded instance of the desired machine learning model; and
selecting the available execution environment to be the execution environment; and
instructing the API endpoint to forward the input data to the selected execution environment to obtain output data that is returned to the client device.
13. The non-transitory computer-readable storage medium of claim 12 , wherein the API endpoint is associated with the desired machine learning model.
14. The non-transitory computer-readable storage medium of claim 13 , wherein the request includes an identifier of the API endpoint.
15. The non-transitory computer-readable storage medium of claim 12 , wherein the determined virtual machine is not being used in conjunction with input data of any other client device.
16. The non-transitory computer-readable storage medium of claim 12 , wherein the input data comprises client identifiers that are associated with client account information.
17. The non-transitory computer-readable storage medium of claim 16 , wherein the client account information comprises one or more from the group of: client location, gender, account details, and previous purchases.
18. The non-transitory computer-readable storage medium of claim 12 , wherein the set of operations further comprises:
preloading a dataset associated with the desired machine learning model into one or more memories that is accessible by the selected execution environment.
19. The non-transitory computer-readable storage medium of claim 12 , wherein the set of operations further comprises generating a notification indicating the selected execution environment is ready for operation.
20. The non-transitory computer-readable storage medium of claim 19 , wherein the notification includes an address of the API endpoint.