Method, electronic device, and computer program product for managing inference process

1 . A method for managing an inference process, the inference process being implemented based on a machine learning model, the method comprising:

determining, in a front-end stage of a processor-based multi-stage scheduler architecture, based on extraction from a computational graph defining the machine learning model, dependency relationships between a set of functions for implementing the inference process, the processor-based multi-stage scheduler architecture further comprising a scheduling stage and a back-end stage that includes a code generator, the scheduling stage being coupled to the front-end stage and the back-end stage;

acquiring, in at least one edge device located in an edge computing network, a set of computing units available to execute the inference process;

determining, in the scheduling stage of the processor-based multi-stage scheduler architecture, the set of computing units of the at least one edge device, at least in part by invoking at least a first application programming interface configured to obtain from the at least one edge device, via at least one multi-access edge computing interface of the at least one edge device, information characterizing available computing units of the at least one edge device;

selecting, in the scheduling stage of the processor-based multi-stage scheduler architecture, at least one computing unit for executing the set of functions from the set of computing units, at least in part by invoking one or more additional application programming interfaces different than the first application programming interface, wherein the scheduling stage is further configured to control the code generator of the back-end stage to generate code for executing the set of functions on the selected at least one computing unit; and

causing the at least one computing unit to execute the set of functions based on the dependency relationships utilizing the code generated by the code generator of the back-end stage.

2 . The method according to claim 1 , wherein selecting the at least one computing unit comprises:

determining, for a function in the set of functions, an operation type of the function; and

selecting, based on the operation type, a computing unit for executing the function from the set of computing units.

3 . The method according to claim 2 , wherein selecting the at least one computing unit further comprises:

acquiring a computing unit configuration describing a mapping between functions of the operation type and computing units; and

selecting, based on the computing unit configuration, a computing unit corresponding to the operation type from the set of computing units.

4 . The method according to claim 1 , wherein selecting the at least one computing unit comprises: selecting, from the set of computing units, different computing units for executing different functions in the set of functions, respectively.

5 . The method according to claim 1 , wherein causing the at least one computing unit to execute the set of functions comprises:

determining, based on the dependency relationships, a plurality of parallel functions in the set of functions that can be executed in parallel;

executing the plurality of parallel functions in parallel using the at least one computing unit; and

executing, based on the dependency relationships, other functions than the plurality of parallel functions in the set of functions serially using the at least one computing unit.

6 . The method according to claim 5 , wherein determining the plurality of parallel functions comprises:

acquiring a parallel algorithm configuration for executing the inference process, the parallel algorithm configuration comprising at least any one of the following: a data parallel algorithm, a model parallel algorithm, and a pipeline parallel algorithm; and

determining the plurality of parallel functions based on the parallel algorithm configuration.

7 . The method according to claim 1 , wherein causing the at least one computing unit to execute the set of functions comprises:

specifying an input queue for storing to-be-processed data associated with the inference process; and

causing the at least one computing unit to execute the inference process for the to-be-processed data in the input queue.

8 . The method according to claim 1 , wherein causing the at least one computing unit to execute the set of functions comprises:

specifying an output queue for storing an inference result generated by the inference process; and

causing the at least one computing unit to store the inference result generated by the inference process to the output queue.

9 . The method according to claim 1 , wherein the method is implemented based on an application programming interface defined in a dynamic link library, and the method is invoked by a main application that uses the inference process.

10 . The method according to claim 1 , wherein the set of computing units comprises a hardware architecture based on at least any one of the following: a central processing unit, a graphics processing unit, a dedicated hardware accelerator, a field programmable gate array, and a system on chip.

11 . An electronic device, comprising:

at least one processor; and

memory coupled to the at least one processor, wherein the memory has instructions stored therein which, when executed by the at least one processor, cause the electronic device to perform actions for managing an inference process, the inference process being implemented based on a machine learning model, and the actions comprising:

determining, in a front-end stage of a processor-based multi-stage scheduler architecture, based on extraction from a computational graph defining the machine learning model, dependency relationships between a set of functions for implementing the inference process, the processor-based multi-stage scheduler architecture further comprising a scheduling stage and a back-end stage that includes a code generator, the scheduling stage being coupled to the front-end stage and the back-end stage;

acquiring, in at least one edge device located in an edge computing network, a set of computing units available to execute the inference process;

determining, in the scheduling stage of the processor-based multi-stage scheduler architecture, the set of computing units of the at least one edge device, at least in part by invoking at least a first application programming interface configured to obtain from the at least one edge device, via at least one multi-access edge computing interface of the at least one edge device, information characterizing available computing units of the at least one edge device;

selecting, in the scheduling stage of the processor-based multi-stage scheduler architecture, at least one computing unit for executing the set of functions from the set of computing units, at least in part by invoking one or more additional application programming interfaces different than the first application programming interface, wherein the scheduling stage is further configured to control the code generator of the back-end stage to generate code for executing the set of functions on the selected at least one computing unit; and

causing the at least one computing unit to execute the set of functions based on the dependency relationships utilizing the code generated by the code generator of the back-end stage.

12 . The electronic device according to claim 11 , wherein selecting the at least one computing unit comprises:

determining, for a function in the set of functions, an operation type of the function; and

selecting, based on the operation type, a computing unit for executing the function from the set of computing units.

13 . The electronic device according to claim 12 , wherein selecting the at least one computing unit further comprises:

acquiring a computing unit configuration describing a mapping between functions of the operation type and computing units; and

selecting, based on the computing unit configuration, a computing unit corresponding to the operation type from the set of computing units.

14 . The electronic device according to claim 11 , wherein selecting the at least one computing unit comprises: selecting, from the set of computing units, different computing units for executing different functions in the set of functions, respectively.

15 . The electronic device according to claim 11 , wherein causing the at least one computing unit to execute the set of functions comprises:

determining, based on the dependency relationships, a plurality of parallel functions in the set of functions that can be executed in parallel;

executing the plurality of parallel functions in parallel using the at least one computing unit; and

executing, based on the dependency relationships, other functions than the plurality of parallel functions in the set of functions serially using the at least one computing unit.

16 . The electronic device according to claim 15 , wherein determining the plurality of parallel functions comprises:

acquiring a parallel algorithm configuration for executing the inference process, the parallel algorithm configuration comprising at least any one of the following: a data parallel algorithm, a model parallel algorithm, and a pipeline parallel algorithm; and

determining the plurality of parallel functions based on the parallel algorithm configuration.

17 . The electronic device according to claim 11 , wherein causing the at least one computing unit to execute the set of functions comprises:

specifying an input queue for storing to-be-processed data associated with the inference process; and

causing the at least one computing unit to execute the inference process for the to-be-processed data in the input queue.

18 . The electronic device according to claim 11 , wherein causing the at least one computing unit to execute the set of functions comprises:

specifying an output queue for storing an inference result generated by the inference process; and

causing the at least one computing unit to store the inference result generated by the inference process to the output queue.

19 . The electronic device according to claim 11 , wherein

the actions are implemented based on an application programming interface defined in a dynamic link library, and the actions are invoked by a main application that uses the inference process; and

the set of computing units comprises a hardware architecture based on at least any one of the following: a central processing unit, a graphics processing unit, a dedicated hardware accelerator, a field programmable gate array, and a system on chip.

20 . A computer program product comprising a non-transitory computer-readable medium having machine-executable instructions stored therein, wherein the machine-executable instructions when executed cause at least one device to perform actions for managing an inference process, the inference process being implemented based on a machine learning model, the actions comprising:

determining, in a front-end stage of a processor-based multi-stage scheduler architecture, based on extraction from a computational graph defining the machine learning model, dependency relationships between a set of functions for implementing the inference process, the processor-based multi-stage scheduler architecture further comprising a scheduling stage and a back-end stage that includes a code generator, the scheduling stage being coupled to the front-end stage and the back-end stage;

acquiring, in at least one edge device located in an edge computing network, a set of computing units available to execute the inference process;

determining, in the scheduling stage of the processor-based multi-stage scheduler architecture, the set of computing units of the at least one edge device, at least in part by invoking at least a first application programming interface configured to obtain from the at least one edge device, via at least one multi-access edge computing interface of the at least one edge device, information characterizing available computing units of the at least one edge device;

selecting, in the scheduling stage of the processor-based multi-stage scheduler architecture, at least one computing unit for executing the set of functions from the set of computing units, at least in part by invoking one or more additional application programming interfaces different than the first application programming interface, wherein the scheduling stage is further configured to control the code generator of the back-end stage to generate code for executing the set of functions on the selected at least one computing unit; and

causing the at least one computing unit to execute the set of functions based on the dependency relationships utilizing the code generated by the code generator of the back-end stage.