Generating and executing context-specific neural network models based on target runtime parameters
The disclosed embodiments relate to a system that generates and executes a deep neural network (DNN) based on target runtime parameters. During operation, the system receives a trained original model and a set of target runtime parameters for the DNN, wherein the target runtime parameters are associated with one or more of the following for the DNN: desired operating conditions, desired resource utilization, and desired accuracy of results. Next, the system generates a context-specific model based on the original model and the set of target runtime parameters. The system also generates an operational plan for executing both the original model and the context-specific model to meet requirements of the target runtime parameters. Finally, the system controls execution of the original model and the context-specific model based on the operational plan.
1. A method for generating and executing a deep neural network (DNN) based on target runtime parameters, comprising:
receiving a trained original model and a set of target runtime parameters for the DNN, wherein the target runtime parameters are associated with one or more of the following for the DNN: desired operating conditions, desired resource utilization, and desired accuracy of results;
generating a context-specific model based on the original model and the set of target runtime parameters;
generating an operational plan for executing both the original model and the context-specific model to meet requirements of the target runtime parameters; and
controlling execution of the original model and the context-specific model based on the operational plan; and
deploying and executing the context-specific model at a location in a hierarchy of computing nodes, wherein the location is determined based on the target runtime parameters.
2. The method of claim 1 , wherein the target runtime parameters are generated based on current operating conditions at the location in a hierarchy of computing nodes.
3. The method of claim 1 , wherein information regarding the target runtime parameters, locations of the original model and the context-specific model in the hierarchy of computing nodes, and results produced by the original model and the context-specific model are stored at a common location in the hierarchy of computing nodes, wherein the information is used to control execution of the operational plan.
4. The method of claim 1 , wherein the operational plan involves executing and obtaining results from the original model when results from the context-specific model do not meet requirements of the target runtime parameters.
5. The method of claim 4 , wherein the original model and the context-specific model are executed on different devices in the hierarchy of computing nodes.
6. The method of claim 1 , wherein the context-specific model is moved to a location in the hierarchy of computing nodes, which is closer to a data source for the context-specific model.
7. The method of claim 1 , wherein the context-specific model is designed to operate within constraints of computing capabilities of a specific device in the hierarchy of computing nodes.
8. The method of claim 1 , wherein the context-specific model performs inference operations that are specific to a particular user, and wherein the original model performs inference operations that are not specific to a particular user.
9. The method of claim 1 , wherein weights in the context-specific model are generated by quantizing corresponding weights in the original model, so that the weights in the context-specific model have a lower bit-precision than the corresponding weights in the original model.
10. The method of claim 1 , wherein the context-specific model is trained to operate over a subset of an input range and/or a subset of an output range of the original model.
11. The method of claim 1 , wherein the context-specific model is smaller than the original model, and wherein the context-specific model is generated through a knowledge distillation technique, which uses the original model to train the context-specific model.
12. The method of claim 1 , wherein the operational plan is generated to achieve one or more of the following:
maximizing classification accuracy of the DNN;
minimizing latency involved in executing the DNN to produce results;
minimizing computational operations performed while executing the DNN; and
minimizing power consumption while executing the DNN.
13. The method of claim 1 , wherein generating the context-specific model involves generating different context-specific models based on different sets of runtime parameters.
14. The method of claim 13 , wherein the operational plan switches among executing different context-specific models as an operating environment of the DNN changes.
15. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for generating and executing a deep neural network (DNN) based on target runtime parameters, the method comprising:
receiving a trained original model and a set of target runtime parameters for the DNN, wherein the target runtime parameters are associated with one or more of the following for the DNN: desired operating conditions, desired resource utilization, and desired accuracy of results;
generating a context-specific model based on the original model and the set of target runtime parameters;
generating an operational plan for executing both the original model and the context-specific model to meet requirements of the target runtime parameters; and
controlling execution of the original model and the context-specific model based on the operational plan; and
deploying and executing the context-specific model at a location in a hierarchy of computing nodes, wherein the location is determined based on the target runtime parameters.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the target runtime parameters are generated based on current operating conditions at the location in a hierarchy of computing nodes.
17. The non-transitory computer-readable storage medium of claim 15 , wherein information regarding the target runtime parameters, locations of the original model and the context-specific model in the hierarchy of computing nodes, and results produced by the original model and the context-specific model are stored at a common location in the hierarchy of computing nodes, wherein the information is used to control execution of the operational plan.
18. The non-transitory computer-readable storage medium of claim 15 , wherein the operational plan involves executing and obtaining results from the original model when results from the context-specific model do not meet requirements of the target runtime parameters.
19. The non-transitory computer-readable storage medium of claim 18 , wherein the original model and the context-specific model are executed on different devices in the hierarchy of computing nodes.
20. The non-transitory computer-readable storage medium of claim 15 , wherein the context-specific model is moved to a location in the hierarchy of computing nodes, which is closer to a data source for the context-specific model.
21. The non-transitory computer-readable storage medium of claim 15 , wherein the context-specific model is designed to operate within constraints of computing capabilities of a specific device in the hierarchy of computing nodes.
22. The non-transitory computer-readable storage medium of claim 15 , wherein the context-specific model performs inference operations that are specific to a particular user, and wherein the original model performs inference operations that are not specific to a particular user.
23. The non-transitory computer-readable storage medium of claim 15 , wherein weights in the context-specific model are generated by quantizing corresponding weights in the original model, so that the weights in the context-specific model have a lower bit-precision than the corresponding weights in the original model.
24. The non-transitory computer-readable storage medium of claim 15 , wherein the context-specific model is trained to operate over a subset of the input range and/or a subset of output range of the original model.
25. The non-transitory computer-readable storage medium of claim 15 , wherein the context-specific model is smaller than the original model, and wherein the context-specific model is generated through a knowledge distillation technique, which uses the original model to train the context-specific model.
26. The non-transitory computer-readable storage medium of claim 15 , wherein the operational plan is generated to achieve one or more of the following:
maximizing classification accuracy of the DNN;
minimizing latency involved in executing the DNN to produce results;
minimizing computational operations performed while executing the DNN; and
minimizing power consumption while executing the DNN.
27. The non-transitory computer-readable storage medium of claim 15 , wherein generating the context-specific model involves generating different context-specific models based on different sets of runtime parameters.
28. The non-transitory computer-readable storage medium of claim 27 , wherein the operational plan switches among executing different context-specific models as an operating environment of the DNN changes.
29. A system that generates and executes a deep neural network (DNN) based on target runtime parameters, comprising:
at least one processor and at least one associated memory; and
a processing mechanism that executes on the at least one processor, wherein during operation, the processing mechanism:
receives a trained original model and a set of target runtime parameters for the DNN, wherein the target runtime parameters are associated with one or more of the following for the DNN: desired operating conditions, desired resource utilization, and desired accuracy of results;
generates a context-specific model based on the original model and the set of target runtime parameters;
generates an operational plan for executing both the original model and the context-specific model to meet requirements of the target runtime parameters;
controls execution of the original model and the context-specific model based on the operational plan; and
deploys and executes the context-specific model at a location in a hierarchy of computing nodes, wherein the location is determined based on the target runtime parameters.