Analytical model to optimize deep learning models
Techniques for optimizing and deploying deep neural network (CNN) machine learning models for inference using static analysis are described. A method includes obtaining a deep neural network (DNN) machine learning (ML) model, generating an intermediate representation for the ML model, the intermediate representation including one or more nodes corresponding to one or more operators utilized by the ML model, identifying, for at least one node of the intermediate representation, an optimized schedule for at least one operator corresponding to the at least one node using a static analysis that is based on a hardware-specific cost model, generating an optimized intermediate representation using the optimized schedule that is optimized for execution on a hardware platform, and generating code corresponding to the ML model based at least in part on the optimized intermediate representation, wherein the code is specific to the hardware platform.
1. A computer-implemented method comprising:
receiving, at an interface of a multi-tenant provider network, a request to deploy a deep neural network (DNN) type machine learning (ML) model to one or more edge computing devices having a hardware platform from among a plurality of hardware platforms, the request including an identifier of the ML model or a storage location of the ML model within the provider network;
obtaining the ML model from the storage location;
generating an intermediate representation for the ML model, the intermediate representation including one or more nodes corresponding to one or more operators utilized by the ML model;
generating a plurality of scores for a plurality of schedules for at least one node of the intermediate representation using a hardware-specific linear cost model, the hardware-specific linear cost model including terms corresponding to different hardware features of the hardware platform;
identifying, for the at least one node of the intermediate representation, an optimized schedule for at least one operator corresponding to the at least one node based on the plurality of scores generated using the hardware-specific linear cost model;
generating an optimized intermediate representation using the optimized schedule;
generating code corresponding to the ML model based at least in part on the optimized intermediate representation, wherein the code is specific to the hardware platform; and
transmitting the code for deployment to the one or more edge computing devices.
2. The computer-implemented method of claim 1 , wherein the hardware-specific linear cost model is a CPU cost model and wherein the terms include at least one of a number of single instruction multiple data (SIMD) instructions, an estimate of L1 cache miss, or instruction level parallelism.
3. The computer-implemented method of claim 1 , wherein the hardware-specific linear cost model is a GPU cost model and wherein the terms include at least one of a number of parallel thread execution (PTX) instructions or instruction level parallelism.
4. A computer-implemented method comprising:
obtaining a deep neural network (DNN) machine learning (ML) model;
generating an intermediate representation for the ML model, the intermediate representation including one or more nodes corresponding to one or more operators utilized by the ML model;
identifying, for at least one node of the intermediate representation, an optimized schedule for at least one operator corresponding to the at least one node using a static analysis that is based on a hardware-specific cost model;
generating an optimized intermediate representation using the optimized schedule that is optimized for execution on a hardware platform from among a plurality of hardware platforms; and
generating code corresponding to the ML model based at least in part on the optimized intermediate representation, wherein the code is specific to the hardware platform.
5. The computer-implemented method of claim 4 , wherein identifying, for the at least one node of the intermediate representation, an optimized schedule for at least one operator corresponding to the at least one node using a static analysis that is based on a hardware-specific cost model, further comprises:
generating a plurality of scores for a plurality of schedules for the at least one node of the intermediate representation using the hardware-specific cost model, the hardware-specific cost model including terms corresponding to different features of the hardware platform; and
identifying the optimized schedule based on the plurality of scores using a parallel black box search algorithm.
6. The computer-implemented method of claim 5 , wherein identifying, for the at least one node of the intermediate representation, an optimized schedule for at least one operator corresponding to the at least one node using a static analysis that is based on a hardware-specific cost model, further comprises:
extracting one or more hardware features based at least on the intermediate representation and the code corresponding to the ML model;
determining a number of hardware instructions based at least on the intermediate representation and the code corresponding to the ML model; and
generating the plurality of scores using the hardware-specific cost model, the one or more hardware features, and the number of hardware instructions.
7. The computer-implemented method of claim 6 , wherein each term of the hardware-specific cost model is associated with a coefficient associated with the hardware platform.
8. The computer-implemented method of claim 6 , wherein determining a number of hardware instructions based at least on the intermediate representation and the code corresponding to the ML model, further comprises:
identifying one or more loop structures in the intermediate representation;
identifying one or more corresponding blocks of the code corresponding to the ML model that match the one or more loop structures; and
counting a number of hardware instructions in the matching one or more corresponding blocks of the code.
9. The computer-implemented method of claim 8 , wherein the hardware instructions include single instruction multiple data (SIMD) assembly code instructions or parallel thread execution (PTX) instructions.
10. The computer-implemented method of claim 6 , wherein extracting one or more hardware features based at least on the intermediate representation and the code corresponding to the ML model further comprises:
identifying a plurality of loop nodes in the intermediate representation;
determining a data movement and data footprint for each loop node from the plurality of loop nodes; and
determining a cache localization feature based on the data movement and data footprint for each loop node.
11. The computer-implemented method of claim 9 , wherein extracting one or more hardware features based at least on the intermediate representation and the code corresponding to the ML model further comprises estimating parallelization efficiency using a parallelization model that simulates instruction level parallelization for each block of the code corresponding to the ML model.
12. The computer-implemented method of claim 9 , wherein extracting one or more hardware features based at least on the intermediate representation and the code corresponding to the ML model further comprises:
determining a workload per thread;
determining a streaming multiprocessor usage; and
determining a single block resource utilization.
13. The computer-implemented method of claim 4 , wherein the ML model is an image classification model, an object detection model, an image segmentation model, or a natural language processing model.
14. A system comprising:
a first one or more electronic devices implementing a storage service; and
a second one or more electronic devices implementing a machine learning service, the machine learning service including instructions that upon execution cause the machine learning service to:
obtain a deep neural network (DNN) machine learning (ML) model from the storage service;
generate an intermediate representation for the ML model, the intermediate representation including one or more nodes corresponding to one or more operators utilized by the ML model;
identify, for at least one node of the intermediate representation, an optimized schedule for at least one operator corresponding to the at least one node using a static analysis that is based on a hardware-specific cost model;
generate an optimized intermediate representation using the optimized schedule that is optimized for execution on a hardware platform from among a plurality of hardware platforms; and
generate code corresponding to the ML model based at least in part on the optimized intermediate representation, wherein the code is specific to the hardware platform.
15. The system of claim 14 , wherein to identify, for the at least one node of the intermediate representation, an optimized schedule for at least one operator corresponding to the at least one node using a static analysis that is based on a hardware-specific cost model, the instructions, when executed, further cause the machine learning service to:
generate a plurality of scores for a plurality of schedules for the at least one node of the intermediate representation using the hardware-specific cost model, the hardware-specific cost model including terms corresponding to different features of the hardware platform; and
identify the optimized schedule based on the plurality of scores using a parallel black box search algorithm.
16. The system of claim 15 , wherein to identify, for the at least one node of the intermediate representation, an optimized schedule for at least one operator corresponding to the at least one node using a static analysis that is based on a hardware-specific cost model, the instructions, when executed, further cause the machine learning service to:
extract one or more hardware features based at least on the intermediate representation and the code corresponding to the ML model;
determine a number of hardware instructions based at least on the intermediate representation and the code corresponding to the ML model; and
generate the plurality of scores using the hardware-specific cost model, the one or more hardware features, and the number of hardware instructions.
17. The system of claim 16 , wherein each term of the hardware-specific cost model is associated with a coefficient associated with the hardware platform.
18. The system of claim 16 , wherein to determine a number of hardware instructions based at least on the intermediate representation and the code corresponding to the ML model, the instructions, when executed, further cause the machine learning service to:
identify one or more loop structures in the intermediate representation;
identify one or more corresponding blocks of the code corresponding to the ML model that match the one or more loop structures; and
count a number of hardware instructions in the matching one or more corresponding blocks of the code.
19. The system of claim 18 , wherein the hardware instructions include single instruction multiple data (SIMD) assembly code instructions or parallel thread execution (PTX) instructions.
20. The system of claim 18 , wherein to extract one or more hardware features based at least on the intermediate representation and the code corresponding to the ML model, the instructions, when executed, further cause the machine learning service to:
identifying a plurality of loop nodes in the intermediate representation;
determining a data movement and data footprint for each loop node from the plurality of loop nodes; and
determining a cache localization feature based on the data movement and data footprint for each loop node.