Granted Patent B2

US 12,572,339 · App. 18/320,975 · Granted Mar 10, 2026

Inventors: Haruki Imai (Yokohama, JP); Yasushi Negishi (Machida, JP); Tung D. Le (Ichikawa, JP); Kiyokuni Kawachiya (Yokohama, JP)

Assignee: International Business Machines Corporation

G06F8/433G06F8/35G06F8/4441G06F8/451G06F9/5033G06F9/5066

Abstract

Claims (78)

1 . A computer program product for accelerating inference performance in an artificial intelligence model, the computer program product comprising:

one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions comprising:

receiving, by a processor, a plurality of operations associated with the artificial intelligence model;

generating, by the processor, a computational graph for the artificial intelligence model;

categorizing, by the processor, each operation of the plurality of operations into one of three categories including:

accelerator designated operations;

central processing unit (CPU) designated operations; and

undetermined processing designated operations;

determining, by the processor, an estimated processing time for each operation of the plurality of operations;

inserting, by the processor, the plurality of operations into the computational graph;

dividing, by the processor, the computational graph into a plurality of sub-graphs;

determining, by the processor, edges of the plurality of sub-graphs where pre-processing steps will be performed; and

converting, by the processor, in one or more sub-graphs of the plurality of sub-graphs, the undetermined processing designated operations, into one of the accelerator designated operations or the CPU designated operations, based on a condition that reduces a number of the pre-processing steps in the one or more sub-graphs of the plurality of sub-graphs, wherein the accelerator designated operations are executed by an accelerator, and wherein the CPU designated operations are executed by a CPU.

2 . The computer program product of claim 1 , wherein the dividing of the computational graph into the plurality of sub-graphs is performed at a point in the computational graph where one of the accelerator designated operations or the CPU designated operations is adjacent to one of the undetermined processing designated operations.

3 . The computer program product of claim 1 , wherein, upon determining that one sub-graph of the plurality of sub-graphs includes vertices that are the accelerator designated operations, the conversion converts all the undetermined processing designated operations in the sub-graph of the plurality of sub-graphs into the accelerator designated operations.

4 . The computer program product of claim 1 , wherein, upon determining that one sub-graph of the plurality of sub-graphs includes vertices that are the CPU designated operations, the conversion converts all the undetermined processing designated operations in the sub-graph of the plurality of sub-graphs into the CPU designated operations.

5 . The computer program product of claim 1 , wherein, upon determining that one sub-graph of the plurality of sub-graphs includes vertices that include one accelerator designated operation and one CPU designated operation, the conversion converts the undetermined processing designated operations in the sub-graph of the plurality of sub-graphs so that only a single one of the accelerator designated operations is adjacent to a single one of the CPU designated operations.

6 . The computer program product of claim 1 , wherein the program instructions further comprise:

selecting a plurality of combinations of the accelerator designated operations and the CPU designated operations for one or more of the undetermined processing designated operations in one sub-graph of the plurality of sub-graphs;

inserting one or more temporary pre-processing steps at the edges of the plurality of sub-graphs, wherein the edges of the plurality of sub-graphs include the accelerator designated operations and the CPU designated operations;

determining an elapsed time for processing through a path of the sub-graph of the plurality of sub-graphs, wherein the path of the sub-graph of the plurality of sub-graphs includes the one or more temporary pre-processing steps and the undetermined processing designated operations using different selected combinations of the accelerator designated operations and the CPU designated operations; and

selecting a combination of the different selected combinations of the accelerator designated operations and the CPU designated operations, that demonstrates a lowest elapsed time.

7 . The computer program product of claim 1 , wherein for the division of the computational graph into the plurality of sub-graphs, the program instructions further comprise:

associating vertices that are either the accelerator designated operations or the CPU designated operations as being ends of a same sub-graph;

determining a computation time from each undetermined processing designated operation to each end in the same sub-graph, wherein the computation time equals a processing time from an individual undetermined processing designated operation to:

one of the accelerator designated operations at the ends of the same sub-graph; and

one of the CPU designated operations at the ends of the same sub-graph; and

determining the edges of the plurality of sub-graphs where the pre-processing steps will be performed based on solving a minimum cut problem using the computation time of the individual undetermined processing designated operation.

8 . A method for accelerating inference performance in an artificial intelligence model, comprising:

receiving, by a processor, a plurality of operations associated with the artificial intelligence model;

generating, by the processor, a computational graph for the artificial intelligence model;

categorizing, by the processor, each operation of the plurality of operations into one of three categories including:

accelerator designated operations;

central processing unit (CPU) designated operations; and

undetermined processing designated operations;

determining, by the processor, an estimated processing time for each operation of the plurality of operations;

inserting, by the processor, the plurality of operations into the computational graph;

dividing, by the processor, the computational graph into a plurality of sub-graphs;

determining, by the processor, edges of the plurality of sub-graphs where pre-processing steps will be performed; and

converting, by the processor, in one or more sub-graphs of the plurality of sub-graphs, the undetermined processing designated operations, into one of the accelerator designated operations or the CPU designated operations, based on a condition that minimizes a number of the pre-processing steps in the one or more sub-graphs of the plurality of sub-graphs, wherein the accelerator designated operations are executed by an accelerator, and wherein the CPU designated operations are executed by a CPU.

9 . The method of claim 8 , wherein the dividing of the computational graph into the plurality of sub-graphs is performed at a point in the computational graph where one of the accelerator designated operations or the CPU designated operations is adjacent to one of the undetermined processing designated operations.

10 . The method of claim 8 , wherein, upon determining that one sub-graph of the plurality of sub-graphs includes vertices that are the accelerator designated operations, the conversion converts all the undetermined processing designated operations in the sub-graph of the plurality of sub-graphs into the accelerator designated operations.

11 . The method of claim 8 , wherein, upon determining that one sub-graph of the plurality of sub-graphs includes vertices that are the CPU designated operations, the conversion converts all the undetermined processing designated operations in the sub-graph of the plurality of sub-graphs into the CPU designated operations.

12 . The method of claim 8 , wherein, upon determining that one sub-graph of the plurality of sub-graphs includes vertices that include one accelerator designated operation and one CPU designated operation, the conversion converts the undetermined processing designated operations in the sub-graph of the plurality of sub-graphs so that only a single one of the accelerator designated operations is adjacent to a single one of the CPU designated operations.

13 . The method of claim 8 , further comprising:

selecting a plurality of combinations of the accelerator designated operations and the CPU designated operations for one or more of the undetermined processing designated operations in one sub-graph of the plurality of sub-graphs;

inserting one or more temporary pre-processing steps at the edges of the plurality of sub-graphs, wherein the edges of the plurality of sub-graphs include the accelerator designated operations and the CPU designated operations;

determining an elapsed time for processing through a path of the sub-graph of the plurality of sub-graphs, wherein the path of the sub-graph of the plurality of sub-graphs includes the one or more temporary pre-processing steps and the undetermined processing designated operations using different selected combinations of the accelerator designated operations and the CPU designated operations; and

selecting a combination of the different selected combinations of the accelerator designated operations and the CPU designated operations, that demonstrates a lowest elapsed time.

14 . The method of claim 8 , wherein for the division of the computational graph into the plurality of sub-graphs, the method further comprises:

associating vertices that are either both the accelerator designated operations or the CPU designated operations as being ends of a same sub-graph;

determining a computation time from each undetermined processing designated operation to each end in the same sub-graph, wherein the computation time equals a processing time from an individual undetermined processing designated operation to:

one of the accelerator designated operations at the ends of the same sub-graph; and

one of the CPU designated operations at the ends of the same sub-graph; and

determining the edges of the plurality of sub-graphs where the pre-processing steps will be performed based on solving a minimum cut problem using the computation time of the individual undetermined processing designated operation.

15 . A computing device configured to accelerate inference performance in an artificial intelligence model, comprising:

a processor; and

a memory coupled to the processor, the memory storing instructions to cause the processor to perform acts comprising:

receiving a plurality of operations associated with the artificial intelligence model;

generating a computational graph for the artificial intelligence model;

categorizing each operation of the plurality of operations into one of three categories including:

accelerator designated operations;

central processing unit (CPU) designated operations; and

undetermined processing designated operations;

determining an estimated processing time for each operation of the plurality of operations;

inserting the plurality of operations into the computational graph;

dividing the computational graph into a plurality of sub-graphs;

determining edges of the plurality of sub-graphs where pre-processing steps will be performed; and

converting, in one or more sub-graphs of the plurality of sub-graphs, the undetermined processing designated operations, into one of the accelerator designated operations or the CPU designated operations, based on a condition that reduces a number of the pre-processing steps in the one or more sub-graphs of the plurality of sub-graphs, wherein the accelerator designated operations are executed by an accelerator, and wherein the CPU designated operations are executed by a CPU.

16 . The computing device of claim 15 , wherein the dividing of the computational graph into the plurality of sub-graphs is performed at a point in the computational graph where one of the accelerator designated operations or the CPU designated operations is adjacent to one of the undetermined processing designated operations.

17 . The computing device of claim 15 , wherein, upon determining that one sub-graph of the plurality of sub-graphs includes vertices that are the accelerator designated operations, the conversion converts all the undetermined processing designated operations in the sub-graph of the plurality of sub-graphs into the accelerator designated operations.

18 . The computing device of claim 15 , wherein, upon determining that one sub-graph of the plurality of sub-graphs includes vertices that are the CPU designated operations, the conversion converts all the undetermined processing designated operations in the sub-graph of the plurality of sub-graphs into the CPU designated operations.

19 . The computing device of claim 15 , wherein, upon determining that one sub-graph of the plurality of sub-graphs includes vertices that include one accelerator designated operation and one CPU designated operation, the conversion converts the undetermined processing designated operations in the sub-graph of the plurality of sub-graphs so that only a single one of the accelerator designated operations is adjacent to a single one of the CPU designated operations.

20 . The computing device of claim 15 , wherein the instructions cause the processor to perform acts further comprising:

selecting a plurality of combinations of the accelerator designated operations and the CPU designated operations for one or more of the undetermined processing designated operations in one sub-graph of the plurality of sub-graphs;

inserting one or more temporary pre-processing steps at the edges of the plurality of sub-graphs, wherein the edges of the plurality of sub-graphs include the accelerator designated operations and the CPU designated operations;

determining an elapsed time for processing through a path of the sub-graph of the plurality of sub-graphs, wherein the path of the sub-graph of the plurality of sub-graphs includes the one or more temporary pre-processing steps and the undetermined processing designated operations using different selected combinations of the accelerator designated operations and the CPU designated operations; and

selecting a combination of the different selected combinations of the accelerator designated operations and the CPU designated operations, that demonstrates a lowest elapsed time.

Assignments (1)

Continuity (1)

References Cited (18)