Granted Patent B2

US 12,475,356 · App. 16/612,361 · Granted Nov 18, 2025

Inventors: Xunyu Chen (Beijing, CN); Qi Guo (Beijing, CN); Jie Wei (Beijing, CN); Linyang Wu (Beijing, CN)

Assignee: CAMBRICON TECHNOLOGIES CORPORATION LIMITED

G06N3/045

Abstract

Claims (122)

1 . A neural network processing method, comprising:

acquiring, by an acquisition unit, a model dataset and model structure parameters of an original neural network,

wherein the model dataset includes network weights corresponding to respective compute nodes of a plurality of compute nodes in the original neural network, and

wherein the model structure parameters include connection values that indicate connections among the plurality of compute nodes in the original neural network;

acquiring, by an operation unit, operational attributes of the respective compute nodes of the plurality of compute nodes in the original neural network,

wherein the operational attributes of the respective compute nodes of the plurality of compute nodes includes a first operational attribute that indicates that instructions corresponding to the plurality of compute nodes are executable on an application specific neural network processor, and a second operational attribute that indicates that the instructions corresponding to the plurality of compute nodes are executable on a general purpose processor,

wherein the application specific neural network processor serves as a coprocessor communicating with the general purpose processor, and

wherein the application specific neural network processor serves as a coprocessor communicating with the general purpose processor;

running, by the operation unit, the original neural network to acquire the instructions that are executable by the application specific neural network processor corresponding to the respective compute nodes of the plurality of compute nodes in the original neural network, according to the model dataset, the model structure parameters, and the operational attributes of the respective compute nodes;

determining, by a control unit, that an operational attribute of a current compute node of the plurality of compute nodes is the first operational attribute; and

based on the determination, storing, by the control unit, network weight and one of the instructions that are executable by the application specific neural network processor without further compilation in a first nonvolatile memory, to generate a first offline model corresponding to the original neural network, wherein,

the instructions are executable by the application specific neural network processor without further compilation,

the stored instruction and network weight correspond to the current compute node,

the first offline model is to be run by the application specific neural network processor without compiling the original neural network,

the storing of the network weight and one of the instructions for the generation of the first offline model corresponding to the original neural network includes:

acquiring a memory allocation manner of the original neural network according to the model dataset and the model structure parameters of the original neural network;

storing related data during running of the original neural network in a volatile memory according to the memory allocation manner, wherein the related data during the running of the original neural network includes the network weights, the instructions, input data, and output data corresponding to the respective compute nodes of the original neural network;

acquiring the network weights and instructions corresponding to the respective compute nodes of the plurality of computing nodes having the first operational attribute in the original neural network from the volatile memory;

storing, in the first nonvolatile memory, the network weights and the instructions corresponding to the respective compute nodes of the plurality of computing nodes having the first operational attribute in the original neural network to generate the first offline model;

acquiring, the network weights and the instructions corresponding to the respective compute nodes of the plurality of computing nodes having the second operational attribute in the original neural network from the volatile memory; and

storing, in a second nonvolatile memory, the network weights corresponding to the respective compute nodes of the plurality of computing nodes having the second operational attribute in the original neural network to generate a second offline model.

2 . The method of claim 1 , wherein the acquisition of the operational attributes of the respective compute nodes of the plurality of compute nodes in the original neural network includes:

determining that the respective compute nodes are executable on the application specific neural network processor, based on the determination that the current compute node is executable on the application specific neural network processor, and

marking the current compute node with the first operational attribute, and based on the determination that the current compute node is executable on the general purpose processor only, marking the current compute node with the second operational attribute.

3 . The method of claim 2 , the determination of whether the respective compute nodes of the plurality of compute nodes are executable on the application specific neural network processor includes:

searching whether the current compute node has an equivalent compute node via a preset function table, wherein the equivalent compute node being a compute node that is executable on the application specific neural network processor,

based on the determination that the current compute node has the equivalent compute node, determining that the current compute node is executable on the application specific neural network processor, and

based on the determination that the current compute node does not have the equivalent compute node, determining that the current compute node is executable merely on the general purpose processor.

4 . The method of claim 2 , wherein the general purpose processor includes one or more of a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), and a field programmable gate array (FGPA), and the second operational attribute includes one or more of a CPU operational attribute, a GPU operational attribute, a DSP operational attribute, and an FGPA attribute.

5 . The method of claim 1 , wherein the acquisition of the operational attributes of the respective compute nodes of the plurality of compute nodes in the original network includes acquiring the operational attributes of the respective compute nodes of the plurality of compute nodes in the original network from the model dataset or the model structure parameters of the original network.

6 . The method of claim 1 , further comprising:

making all first compute nodes among the plurality of compute nodes more than two second compute nodes of the plurality of compute nodes, which are executed in order, equivalent to a first offline node, wherein

the first compute nodes are compute nodes having the first operational attribute,

the two second compute nodes of the plurality of compute nodes are compute nodes having the second operational attribute, and

the first offline model further includes interface data among the first offline nodes and the more than two second compute nodes.

7 . The method of claim 1 , further comprising:

if the operational attribute of the current compute node is the second operational attribute, storing the network weight and the instruction corresponding to a current compute node of the plurality of compute nodes in the second nonvolatile memory to acquire a second offline model corresponding to the original neural network;

wherein the second offline model can include a plurality of second offline sub-models; and

wherein the second offline sub-models include instructions and network weights corresponding to compute nodes having a CPU operational attribute, a GPU operational attribute, or a DSP operational attribute.

8 . The method of claim 7 , further comprising:

making all of second compute nodes among the plurality of compute modes more than two of first compute nodes of the plurality of compute modes, which are executed in order, equivalent to a second offline node, wherein

the first compute nodes are compute nodes having the first operational attribute,

the second compute nodes are compute nodes having the second operational attribute, and

the second offline model further includes interface data among the second offline nodes and the first computer nodes.

9 . The method of claim 1 , while running the original neural network to acquire the instructions corresponding to the respective compute nodes in the original neural network according to the model dataset, the model structure parameters, and the operational attributes of the respective compute node includes:

acquiring order of execution of the respective compute nodes in the original neural network according to the model structure parameter of the original neural network;

acquiring target processors corresponding to the respective compute nodes according to the operational attributes of the respective compute nodes; and

executing the respective compute nodes via the target processors corresponding to the respective compute nodes, according to the order of execution of the respective compute nodes in the original neural network, to acquire the instructions corresponding to the respective compute nodes in the original neural network.

10 . A neural network processing method, comprising:

acquiring, by an acquisition unit, a model dataset and model structure parameters of an original neural network, wherein

the model dataset comprising network weights corresponding to respective compute nodes of a plurality of compute nodes in the original neural network, and

the model structure parameters comprising connections among the plurality of compute nodes in the original network;

making all of first compute nodes among the plurality of compute nodes more than two second compute nodes of the plurality of compute nodes, which are executed in order, equivalent to a first offline node to acquire an equivalent network corresponding to the original network according to the connections among the plurality of compute nodes in the original network, wherein

the first compute nodes being compute nodes having a first operational attribute,

the second compute nodes being compute nodes having a second operational attribute,

the first operational attribute indicates instructions corresponding to the plurality of compute nodes are executable on an application specific neural network processor, and

the second operational attribute that indicates that the instructions corresponding to the plurality of compute nodes are executable on a general purpose processor;

storing, by a control unit, the network weights and one of instructions that are executable by the application specific neural network processor without further compilation in a first nonvolatile memory, to generate a first offline model corresponding to the original neural network, wherein the storing of the network weights and one of the instructions for the generation of the first offline model corresponding to the original neural network includes:

acquiring a memory allocation manner of the original neural network according to the model dataset and the model structure parameters of the original neural network;

storing related data during running of the original neural network in a volatile memory according to the memory allocation manner, wherein the related data during the running of the original neural network includes the network weights, the instructions, input data, and output data corresponding to the respective compute nodes of the original neural network;

acquiring the network weights and instructions corresponding to the respective compute node of the plurality of computing nodes having the first operational attribute in the original neural network from the volatile memory;

storing, in the first nonvolatile memory, the network weights and the instructions corresponding to the respective compute nodes of the plurality of computing nodes having the first operational attribute in the original neural network to generate the first offline model;

acquiring, the network weights and the instructions corresponding to the respective compute nodes of the plurality of computing nodes having the second operational attribute in the original neural network from the volatile memory; and

storing, in a second nonvolatile memory, the network weights corresponding to the respective compute nodes of the plurality of computing nodes having the second operational attribute in the original neural network to generate a second offline model;

if a current compute node in the equivalent network is a first offline node, acquiring, by the acquisition unit, the first offline model from the first nonvolatile memory; and

executing, by an operation unit, the first offline node according to the first offline model, wherein the first offline model comprises network weights and the instructions that are executable on the application specific neural network processor without further compilation corresponding to all of the first compute nodes in the original neural network, wherein

the first offline model is run by an application specific neural network processor without compiling the original neural network, and

the instructions indicate kinds of computation operation to be executed by the respective compute node of the plurality of compute nodes.

11 . The method of claim 10 , wherein the first offline model further includes interface data among the first offline node and the second compute nodes.

12 . The method of claim 10 , further comprising: if the current compute node in the equivalent network is not the first offline node, acquiring a second offline model from the second nonvolatile memory, and executing the current compute node in the equivalent network according to the second offline model.

13 . A neural network processing method, comprising:

acquiring, by an acquisition unit, a model dataset and model structure parameters of an original neural network, wherein

the model dataset comprises network weights corresponding to respective compute nodes of a plurality of compute nodes in the original neural network, and

the model structure parameters comprise connections among the plurality of compute nodes and computational attributes of the respective compute nodes of the plurality of compute nodes s in the original neural network;

acquiring, by an operation unit, operational attributes of the respective compute nodes of the plurality of compute nodes in the original neural network,

wherein the operational attributes of the respective compute nodes of a plurality of compute nodes include a first operational attribute that indicates that instructions corresponding to the plurality of compute nodes executable on an application specific neural network processor, and a second operational attribute that indicates that the instructions corresponding to the compute nodes can only be executed on a general purpose processor,

wherein the application specific neural network processor serves as a coprocessor communicating with the general purpose processor;

running, by the operation unit, the original neural network according to the model dataset and the model structure parameters of the original neural network to acquire instructions that are executable by the application specific neural network processor without further compilation corresponding to the respective compute nodes of a plurality of compute nodes in the original neural network; and

generating, by a control unit, a first offline model corresponding to the original neural network according to the network weights and the instructions that are executable on the application specific neural network processor without further compilation corresponding to the respective compute nodes of the original neural network, wherein

the generating of the first offline model corresponding to the original neural network according to the network weights and the instructions corresponding to the respective compute nodes of the plurality of computing nodes of the original neural network includes:

acquiring a memory allocation manner of the original neural network according to the model dataset and the model structure parameters of the original neural network;

storing related data during running of the original neural network in a volatile memory according to the memory allocation manner, wherein the related data during the running of the original neural network includes the network weights, the instructions, input data, and output data corresponding to the respective compute nodes of the original neural network;

acquiring the network weights and instructions corresponding to the respective compute nodes of the plurality of computing nodes having the first operational attribute in the original neural network from the volatile memory; and

storing, in the first nonvolatile memory, the network weights and the instructions corresponding to the respective compute nodes of the plurality of computing nodes having the first operational attribute in the original neural network to generate the first offline model; and

storing, the first nonvolatile memory, the network weights and the instructions corresponding to the respective compute nodes of the plurality of computing nodes having the first operational attribute in the original neural network to generate the first offline model, wherein the offline model is executable by the application specific neural network processor without compiling the original neural network;

acquiring, the network weights and the instructions corresponding to the respective compute nodes of the plurality of computing nodes having the second operational attribute in the original neural network from the volatile memory; and

storing, in a second nonvolatile memory, the network weights corresponding to the respective compute nodes of the plurality of computing nodes having the second operational attribute in the original neural network to generate a second offline model.

14 . The method of claim 13 , wherein running the original neural network according to the model dataset and the model structure parameters of the original neural network to acquire the instructions corresponding to the respective compute nodes in the original neural network includes:

acquiring, by the acquisition unit, order of execution of the respective compute nodes in the original neural network according to the model structure parameters of the original neural network, and

running, by the operation unit, the original neural network according to the order of execution of the respective compute node sin the original neural network to acquire the instructions corresponding to the respective compute nodes in the original neural network.

15 . The method of claim 13 , wherein the first offline model further includes node interface data which is configured to represent connections among the compute nodes of the original neural network.

16 . The method of claim 13 , wherein

running the original neural network according to the model dataset and the model structure parameters of the original neural network includes:

running, by a processor or a virtual device of a computer system, the original neural network according to the model dataset and the model structure parameters of the original neural network.

17 . The method of claim 13 , further comprising:

acquiring a model dataset and model structure parameters of a new original neural network;

if the new original neural network has a corresponding offline model, acquiring the offline model corresponding to the new original neural network from the nonvolatile memory, and

running the new original neural network according to the offline model corresponding to the new original neural network; and

if the new original neural network does not have a corresponding offline model, running the new original neural network according to the model dataset and the model structure parameters of the new original neural network,

generating and storing an offline model corresponding to the new original neural network in the nonvolatile memory.

18 . A neural network processing method, comprising:

acquiring, by an acquisition unit, model structure parameters of an original neural network, wherein the model structure parameters comprise connections among a plurality of compute nodes in the original neural network;

acquiring, by the acquisition unit, an offline model corresponding to the original neural network from a nonvolatile memory, wherein the offline model corresponding to the original neural network comprises network weights and instructions that are executable on an application specific neural network processor without further compilation corresponding to the respective compute nodes of the original neural network;

acquiring, by an operation unit, operational attributes of the respective compute nodes of the plurality of compute nodes in the original neural network,

wherein the operational attributes of the respective compute nodes include a first operational attribute that indicates that instructions corresponding to the compute nodes executable on the application specific neural network processor, and a second operational attribute that indicates that the instructions corresponding to the compute nodes can executable on a general purpose processor, wherein the application specific neural network processor serves as a coprocessor communicating with the general purpose processor;

determining, by a control unit, that an operational attribute of a current compute node of the plurality of compute nodes is the first operational attribute; and

based on the determination, storing, by the control unit, network weight and one of the instructions that are executable by the application specific neural network processor without further compilation in a first nonvolatile memory, to generate a first offline model corresponding to the original neural network, wherein,

the instructions are executable by the application specific neural network processor without further compilation,

the stored instruction and network weight correspond to the current compute node,

the first offline model is to be run by the application specific neural network processor without compiling the original neural network,

the storing of the network weight and one of the instructions for the generation of the first offline model corresponding to the original neural network includes:

acquiring a memory allocation manner of the original neural network according to the model dataset and the model structure parameters of the original neural network;

storing related data during running of the original neural network in a volatile memory according to the memory allocation manner, wherein the related data during the running of the original neural network includes the network weights, the instructions, input data, and output data corresponding to the respective compute nodes of the original neural network;

acquiring the network weights and instructions corresponding to the respective compute nodes of the plurality of computing nodes having the first operational attribute in the original neural network from the volatile memory;

storing, in the first nonvolatile memory, the network weights and the instructions corresponding to the respective compute nodes of the plurality of computing nodes having the first operational attribute in the original neural network to generate the first offline model;

acquiring, the network weights and the instructions corresponding to the respective compute nodes of the plurality of computing nodes having the second operational attribute in the original neural network from the volatile memory; and

storing, in a second nonvolatile memory, the network weights corresponding to the respective compute nodes of the plurality of computing nodes having the second operational attribute in the original neural network to generate a second offline model;

running, by the operation unit, the original neural network according to the first offline model corresponding to the original neural network and the model structure parameters of the original neural network, wherein the first offline model is run by the application specific neural network processor without compiling the original neural network, wherein the running of the original neural network includes:

acquiring, by the acquisition unit, a third offline model corresponding to a new original neural network from the first nonvolatile memory;

acquiring, by the acquisition unit, order of execution of the respective compute nodes in the new original neural network according to the model structure parameters of the new original neural network;

acquiring, by the acquisition unit, network weights and instructions corresponding to the respective compute nodes in the new original neural network successively from the third offline model according to the order of execution of the respective compute nodes in the new original neural network; and

running, by the operation unit, the new original neural network directly according to the network weights and the instructions corresponding to the respective compute nodes in the new original neural network without repeating the compiling of the respective compute nodes.

Assignments (1)

Priority Claims (2)

Continuity (1)

References Cited (31)