Methods and apparatuses for executing tasks, storage mediums, and electronic devices

1. A method for executing a task, wherein the method is applied to a computing cluster system, and the method comprises:

receiving, by a master computing device node in the computing cluster system, a task code of a to-be-executed task;

dividing, by the master computing device node, the to-be-executed task into subtasks, and for each of the subtasks, determining, by the master computing device node and based on the task code, operators required to execute the subtask, and data transmission relationships between the subtask and other subtasks;

respectively distributing, by the master computing device node, the subtasks to computing nodes in the computing cluster system, such that for each of the computing nodes, the computing node generates an executable task subgraph for the computing node based on the operators required to execute the subtask distributed to the computing node and data transmission relationships between the operators required to execute the subtask distributed to the computing node, wherein the computing nodes are communicatedly connected with the master computing device node; and

running, by the computing nodes, the executable task subgraphs to execute the to-be-executed task;

wherein, respectively distributing, by the master computing device node, the subtasks to the computing nodes in the computing cluster system, such that for each of the computing nodes, the computing node generates the executable task subgraph for the computing node based on the operators required to execute the subtask distributed to the computing node and the data transmission relationships between the operators required to execute the subtask distributed to the computing node comprises:

respectively distributing, by the master computing device node, the subtasks to the computing nodes, such that for each of the computing nodes, the computing node determines a target subtask based on the data transmission relationships between each of the subtasks and other subtasks, wherein the target subtask has a data transmission relationship with the subtask distributed to the computing node; and

determining, by the computing node, whether a data transmission attribute of the computing node when executing the subtask distributed to the computing node is the same as a data transmission attribute of a computing node executing the target subtask;

in response to determining that the data transmission attribute of the computing node when executing the subtask distributed to the computing node is the same as the data transmission attribute of a computing node executing the target subtask, generating, by the computing node, the executable task subgraph for the computing node based on the operators required to execute the subtask distributed to the computing node, the data transmission relationships between the operators required to execute the subtask distributed to the computing node, and the data transmission relationships between the subtask distributed to the computing node and other subtasks;

wherein the data transmission attribute comprises a broadcast attribute, a segmentation attribute, and a local decomposition attribute.

2. The method according to claim 1 , wherein

dividing, by the master computing device node, the to-be-executed task into the subtasks, and for each of the subtasks, determining, by the master computing device node and based on the task code, the operators required to execute the subtask comprises:

determining, by the master computing device node, a global calculating graph of the to-be-executed task based on the task code, wherein the global calculating graph comprises operators required to execute the to-be-executed task and data transmission relationships between the operators required to execute the to-be-executed task; and

dividing, by the master computing device node, the global calculating graph into subtask calculating graphs to divide the to-be-executed task into the subtasks, wherein each of the subtask calculating graphs comprises at least some of the operators in the global calculating graph; and

respectively distributing, by the master computing device node, the subtasks to the computing nodes in the computing cluster system, such that for each of the computing nodes, the computing node generates the executable task subgraph for the computing node based on the operators required to execute the subtask distributed to the computing node and the data transmission relationships between the operators required to execute the subtask distributed to the computing node comprises:

respectively distributing, by the master computing device node, the subtask calculating graphs to the computing nodes, such that for each of the computing nodes, the computing node generates the executable task subgraph for the computing node based on the operators of the subtask calculating graph distributed to the computing node and data transmission relationships between the operators of the subtask calculating graph distributed to the computing node.

3. The method according to claim 2 , wherein the method further comprises:

for each of the computing nodes,

determining, by the computing node, an executing order of the operators in the subtask calculating graph distributed to the computing node based on the operators in the subtask calculating graph distributed to the computing node and the data transmission relationships between the operators in the subtask calculating graph distributed to the computing node;

determining, by the computing node, memory-unit-block information required to execute the subtask corresponding to the subtask calculating graph distributed to the computing node, wherein the memory-unit-block information comprises identification information of the memory unit block, identification information of the subtask using the memory unit block, and information of data transmission stored in the memory unit block; and

generating, by the computing node, an executable task subgraph for the computing node based on the executing order and the memory-unit-block information.

4. The method according to claim 1 , wherein

determining, by the master computing device node and based on the task code, the operators required to execute the subtask comprises:

determining, by the master computing device node and based on the task code, the operators required to execute the subtask, and determining, by the master computing device node and based on the task code, data transmission relationships between the subtask and other subtasks; and

respectively distributing, by the master computing device node, the subtasks to the computing nodes in the computing cluster system, such that for each of the computing nodes, the computing node generates the executable task subgraph for the computing node based on the operators required to execute the subtask distributed to the computing node and the data transmission relationships between the operators required to execute the subtask distributed to the computing node comprises:

respectively distributing, by the master computing device node, the subtasks to the computing nodes in the computing cluster system, such that for each of the computing nodes, the computing node generates the executable task subgraph for the computing node based on the operators required to execute the subtask distributed to the computing node, the data transmission relationships between the operators required to execute the subtask distributed to the computing node, and the data transmission relationships between the subtask distributed to the computing node and other subtasks.

5. The method according to claim 1 , wherein the method further comprises:

in response to determining that the data transmission attribute of the computing node when executing the subtask distributed to the computing node is different from the data transmission attribute of a computing node executing the target subtask, transmitting, by the computing node, abnormal information indicating abnormal data transmission between the computing node and the computing node executing the target subtask to the master computing device node, such that the master computing device node generates a transformation subgraph based on the abnormal information returned by one or more computing nodes, wherein the transformation subgraph is configured to transform a data transmission attribute of a computing node to make the data transmission attribute of the computing node same as a data transmission attribute of another computing node; and

generating, by the computing node, the executable task subgraph for the computing node based on the operators required to execute the subtask distributed to the computing node, the data transmission relationships between the operators required to execute the subtask distributed to the computing node, and the transformation subgraph.

6. The method according to claim 5 , wherein

generating, by the master computing device node, the transformation subgraph based on the abnormal information returned by the one or more computing nodes comprises:

generating, by the master computing device node, the transformation subgraph based on the abnormal information returned by the one or more computing nodes and transmitting, by the master computing device node, the transformation subgraph to the one or more computing nodes;

for each of the one or more computing nodes, based on information of data transmission between the computing node and other computing nodes through the transformation subgraph when the computing node executes the subtask distributed to the computing node in the transformation subgraph, determining, by the master computing device node, memory-unit-block information required for data transmission between the computing node and other computing nodes through the transformation subgraph when the computing device executes the subtask distributed to the computing node, as target memory-unit-block information; and

determining, by the master computing device node, an identification number of the target memory-unit-block information, and storing, by the master computing device node, the identification number of the target memory-unit-block information and an identification number of the computing node as an identification number pair in the transformation subgraph; and

generating, by the computing node, the executable task subgraph for the computing node based on the operators required to execute the subtask distributed to the computing node, the data transmission relationships between the operators required to execute the subtask distributed to the computing node, and the transformation subgraph comprises:

determining, by the computing node, whether there is the identification number pair corresponding to the identification number of the computing node in the transformation subgraph; and

in response to determining that there is the identification number pair corresponding to the identification number of the computing node in the transformation subgraph, obtaining, by the computing node, the target memory-unit-block information based on the identification number of the target memory-unit-block information in the identification number pair corresponding to the identification number of the computing node, and generating, by the computing node, the executable task subgraph for the computing node based on the operators required to execute the subtask distributed to the computing node, the data transmission relationships between the operators required to execute the subtask distributed to the computing node, the transformation subgraph, and the target memory-unit-block information.