Benchmarking machine learning models via performance feedback
Techniques for benchmarking a machine learning model/algorithm are described. For example, in some instances a method includes generating an execution plan for benchmarking of at least one task corresponding to a machine learning model based on an identified machine learning model, identified training data, and at least one objective for the benchmarking job; receiving execution statistics about the execution of the task as a part of the benchmarking job according to the execution plan; and updating the execution plan based at least in part on the received execution statistics of the task.
1. A computer-implemented method comprising:
receiving a request to generate an execution plan for a benchmarking job of at least one training or inference task for an identified machine learning model, the request including at least one objective for the benchmarking job;
generating the execution plan for benchmarking job the at least one training or inference task, the execution plan corresponding to a machine learning model based on the identified machine learning model, identified training data, and the at least one objective for the benchmarking job;
providing the generated execution plan to a monitor service that is to receive statistics about an execution of the execution plan;
receiving execution statistics about the execution of the at least one training or inference task performed by the machine learning model as a part of the benchmarking job according to the execution plan from the monitor service; and
updating the execution plan based at least in part on the received execution statistics about the execution of the at least one training or inference task.
2. The computer-implemented method of claim 1 , further comprising:
triggering execution of the execution plan by the monitor service.
3. The computer-implemented method of claim 1 , wherein the generating of the execution plan is further based on at least one of hardware available to a user requesting the benchmarking job, available jobs to batch with the task, previous execution of the task to be run as a job, and previous execution of similar jobs.
4. A computer-implemented method comprising:
generating an execution plan for a benchmarking job of at least one training or inference task corresponding to a machine learning model based on an identified machine learning model, identified training data, and at least one objective for the benchmarking job;
receiving execution statistics about the execution of the at least one training or inference task performed by the machine learning model as a part of the benchmarking job according to the execution plan; and
updating the execution plan based at least in part on the received execution statistics of the at least one training or inference task.
5. The computer-implemented method of claim 4 , further comprising:
triggering execution of the execution plan by a monitor service.
6. The computer-implemented method of claim 4 , wherein the generating of the execution plan is further based on at least one of hardware available to a user requesting the benchmarking job, available jobs to batch with the task, previous execution of the task to be run as a job, and previous execution of similar jobs.
7. The computer-implemented method of claim 4 , wherein the generating of the execution plan includes simulating the training of a model of the task to determine what hardware is capable of executing the task and meet the objective.
8. The computer-implemented method of claim 4 , wherein the generating of the execution plan includes executing the training of a model of the task to determine what hardware is capable of executing the task and meet the objective.
9. The computer-implemented method of claim 4 , wherein the execution of the task as a part of the benchmarking job according to the execution plan occurs in a virtual machine.
10. The computer-implemented method of claim 4 , wherein the execution of the task as a part of the benchmarking job according to the execution plan occurs in a container.
11. The computer-implemented method of claim 4 , wherein the objective is at least one of: a shortest possible time to complete the task, a minimum hardware usage possible to complete the task, and accuracy of the machine learning model.
12. The computer-implemented method of claim 4 , wherein the generated execution plan is a batch that includes other jobs to be executed on the same hardware.
13. The computer-implemented method of claim 4 , further comprising:
generating metrics about the execution of the task based on the received execution statistics.
14. The computer-implemented method of claim 4 , further comprising:
generating an alarm when the execution of the task exceeds a threshold metric.
15. A system comprising:
execution resources to implement a benchmarking job according to an execution plan; and
a benchmarking service implemented by a second one or more electronic devices, the benchmarking service including instructions that upon execution cause the benchmarking service to:
generate an execution plan for a benchmarking job of at least one training or inference task corresponding to a machine learning model based on an identified machine learning model, identified training data, and at least one objective for the benchmarking job;
receive execution statistics about the execution of the at least one training or inference task performed by the machine learning model as a part of the benchmarking job according to the execution plan; and
update the execution plan based at least in part on the received execution statistics of the at least one training or inference task.
16. The system of claim 15 , wherein the instructions upon execution further cause the benchmarking service to:
trigger execution of the execution plan on the execution resources.
17. The system of claim 15 , wherein the generation of the execution plan is further based on at least one of hardware available to a user requesting the benchmarking job, available jobs to batch with the task, previous execution of the task to be run as a job, and previous execution of similar jobs.
18. The system of claim 15 , wherein the generation of the execution plan includes executing the training of a model of the task to determine what hardware is capable of executing the task and meet the objective.
19. The system of claim 15 , wherein the execution of the task as a part of the benchmarking job according to the execution plan occurs in a virtual machine.
20. The system of claim 15 , wherein the execution of the task as a part of the benchmarking job according to the execution plan occurs in a container.