Scheduling ML services and models with heterogeneous resources

1 . A computer-implemented method, comprising:

determining a timing matrix corresponding to inference times taken for a plurality of machine learning (ML) models to be executed by a plurality of processing resources of a computing device, wherein the plurality of processing resources comprises at least a first and a second type of processing resources;

applying a service-specific model-first scheduling scheme or a service-specific hardware-first scheduling scheme based on the timing matrix to obtain corresponding service-specific mappings;

determining a best mapping from the corresponding service-specific mappings;

scheduling each of the plurality of ML models to a corresponding processing resource from the plurality of processing resources according to the best mapping; and

executing the plurality of ML models using corresponding mapped processing resources.

2 . The method of claim 1 , further comprising:

applying a multi-service model first scheduling scheme or a multi-service hardware first scheduling scheme to obtain corresponding multi-service mappings; and

determining the best mapping from the corresponding service-specific mappings and the corresponding multi-service mappings.

3 . The method of claim 1 , wherein applying a service-specific model-first scheduling scheme comprises applying a model-first scheduling scheme limiting the plurality of ML models to a particular ML service, and wherein applying a multi-service model-first scheduling scheme comprises applying the model-first scheduling scheme limiting the plurality of ML models to a plurality of ML services.

4 . The method of claim 3 , wherein applying the model-first scheduling scheme comprises:

determining a first model that has a worst inference time among the plurality of ML models based on the timing matrix;

determining a first processing resource executing the first model that has a best inference time among different types of processing resources;

mapping the first model to the first processing resource;

indicating the first model and the first processing resource have been mapped;

determining a second model that has a next worst inference time among the plurality of ML models; and

determining a second processing resource to be mapped to the second model that has a best inference time among the different types of processing resources.

5 . The method of claim 1 , wherein applying a service-specific hardware-first scheduling scheme comprises applying a hardware-first scheduling scheme limiting the plurality of ML models to a particular ML service, and wherein applying a multi-service hardware-first scheduling scheme comprises applying the hardware-first scheduling scheme limiting the plurality of ML models to a plurality of services.

6 . The method of claim 5 , wherein applying the hardware-first scheduling scheme comprises:

determining a first processing resource that has a worst inference time among different types of processing resources based on the timing matrix;

determining a first model executed by the first processing resource that has a best inference time among the plurality of ML models;

mapping the first model to the first processing resource;

indicating the first model and the first processing resource have been mapped;

determining a second processing resource that has a next worst inference time among the different types of processing resources; and

determining a second model to be mapped to the second processing resource that has a best inference time among the plurality of ML models.

7 . The method of claim 1 , wherein the plurality of processing resources includes a central processing unit (CPU), graphic processing unit (GPU), or vision processing unit (VPU).

8 . The method of claim 1 , wherein the computing device is an edge device, a smart phone, or an internet of things (IoT) device.

9 . The method of claim 1 , wherein a plurality of ML services comprises face detection, person detection, or vehicle detection services.

10 . The method of claim 1 , wherein face detection comprises age/gender recognition, emotions recognition, facial landmarks, head pose estimation, or person attributes recognition, wherein person detection comprises person attribution recognition and person re-identification, wherein vehicle detection comprises license plate recognition and vehicle attribute recognition ML models.

11 . The method of claim 1 , wherein a default mapping allocates each of the plurality of ML models with the plurality of processing resources in a default order of CPU, GPU, and VPU, according to their availability.

12 . A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations, the operations comprising:

determining a timing matrix corresponding to inference times taken for a plurality of machine learning (ML) models to be executed by a plurality of processing resources of a computing device, wherein the plurality of processing resources comprises at least a first and a second type of processing resources;

applying a service-specific model-first scheduling scheme or a service-specific hardware-first scheduling scheme based on the timing matrix to obtain corresponding service-specific mappings;

determining a best mapping from the corresponding service-specific mappings;

scheduling each of the plurality of ML models to a corresponding processing resource from the plurality of processing resources according to the best mapping; and

executing the plurality of ML models using corresponding mapped processing resources.

13 . The non-transitory machine-readable medium of claim 12 , wherein the operations further comprise:

applying a multi-service model first scheduling scheme or a multi-service hardware first scheduling scheme to obtain corresponding multi-service mappings; and

determining the best mapping from the corresponding service-specific mappings and the corresponding multi-service mappings.

14 . The non-transitory machine-readable medium of claim 12 , wherein applying a service-specific model-first scheduling scheme comprises applying a model-first scheduling scheme limiting the plurality of ML models to a particular ML service, and wherein applying a multi-service model-first scheduling scheme comprises applying the model-first scheduling scheme limiting the plurality of ML models to a plurality of ML services.

15 . The non-transitory machine-readable medium of claim 14 , wherein applying the model-first scheduling scheme comprises:

determining a first model that has a worst inference time among the plurality of ML models based on the timing matrix;

determining a first processing resource executing the first model that has a best inference time among different types of processing resources;

mapping the first model to the first processing resource;

indicating the first model and the first processing resource have been mapped;

determining a second model that has a next worst inference time among the plurality of ML models; and

determining a second processing resource to be mapped to the second model that has a best inference time among the different types of processing resources.

16 . The non-transitory machine-readable medium of claim 12 , wherein applying a service-specific hardware-first scheduling scheme comprises applying a hardware-first scheduling scheme limiting the plurality of ML models to a particular ML service, and wherein applying a multi-service hardware-first scheduling scheme comprises applying the hardware-first scheduling scheme limiting the plurality of ML models to a plurality of ML services.

17 . The non-transitory machine-readable medium of claim 16 , wherein applying the hardware-first scheduling scheme comprises:

determining a first processing resource that has a worst inference time among different types of processing resources based on the timing matrix;

determining a first model executed by the first processing resource that has a best inference time among the plurality of ML models;

mapping the first model to the first processing resource;

indicating the first model and the first processing resource have been mapped;

determining a second processing resource that has a next worst inference time among the different types of processing resources; and

determining a second model to be mapped to the second processing resource that has a best inference time among the plurality of ML models.

18 . A data processing system, comprising:

a processor; and

a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations, the operations including

determining a timing matrix corresponding to inference times taken for a plurality of machine learning (ML) models to be executed by a plurality of processing resources of a computing device, wherein the plurality of processing resources comprises at least a first and a second type of processing resources;

applying a service-specific model-first scheduling scheme or a service-specific hardware-first scheduling scheme based on the timing matrix to obtain corresponding service-specific mappings;

determining a best mapping from the corresponding service-specific mappings;

scheduling each of the plurality of ML models to a corresponding processing resource from the plurality of processing resources according to the best mapping; and

executing the plurality of ML models using corresponding mapped processing resources.

19 . The data processing system of claim 18 , wherein the operations further comprise:

applying a multi-service model first scheduling scheme or a multi-service hardware first scheduling scheme to obtain corresponding multi-service mappings; and

determining the best mapping from the corresponding service-specific mappings and the corresponding multi-service mappings.

20 . The data processing system of claim 18 , wherein applying a service-specific model-first scheduling scheme comprises applying a model-first scheduling scheme limiting the plurality of ML models to a particular ML service, and wherein applying a multi-service model-first scheduling scheme comprises applying the model-first scheduling scheme limiting the plurality of ML models to a plurality of ML services.