Apparatus and method to collect memory dump information of a faulty node in a parallel computer system
An apparatus includes nodes each configured to relay data between the nodes. When a failure occurs in a first-node, a management-node determines, based on power consumption and/or memory usage of the nodes, a collection-node that transmits an instruction in first direction approaching the first-node and a second direction approaching a storage-node, respectively. A second-node that is neither an adjacent-node adjacent to the first-node nor the storage-node, upon receiving data including the instruction, transmit data obtained by adding an evaluation value for the second-node to the received data, in the first or second direction. Each of the adjacent-node and the storage-node, upon receiving data including the instruction, transmit data including an evaluation value for another node included in the received data and an evaluation value for the each node, to the collection-node which determines transmission routes between the collection-node and the first-node, and between the collection-node and the storage-node.
1. A parallel processing apparatus comprising:
a plurality of nodes each including a memory and a processor configured to relay data between the plurality of nodes, wherein
the processor of a management node that is included in the plurality of nodes and manages the plurality of nodes, is configured to:
when a failure occurs in a first node of the plurality of nodes, determine a collection node that is included in the plurality of nodes and collects first information stored in the memory of the first node, based on power consumption and/or memory usage of each of the plurality of nodes, and
transmit, to the determined collection node, data including a first instruction for causing a storage node in the plurality of nodes to store second information generated from the first information into the memory of the storage node;
the processor of the collection node is configured to, upon receiving the first instruction, transmit data including a second instruction that Instructs transmission of an evaluation value for communication performance of each node, in a first direction approaching the first node and in a second direction approaching the storage node, respectively;
the processor of a second node that is included in the plurality of nodes and is neither an adjacent node adjacent to the first node nor the storage node, is configured to:
upon receiving data that is transmitted in the first direction and includes the second instruction, transmit data obtained by adding an evaluation value for communication performance of the second node to the received data, in the first direction, and
upon receiving data that is transmitted in the second direction and includes the second instruction, transmit data obtained by adding an evaluation value for communication performance of the second node to the received data, in the second direction;
the processor of each of the adjacent node and the storage node is configured to, upon receiving data that is transmitted in the first direction or in the second direction and includes the second instruction, transmit data including an evaluation value for communication performance of another node that is included in the received data and an evaluation value for communication performance of the each of the adjacent node and the storage node, to the collection node; and
the processor of the collection node is configured to:
upon receiving data including an evaluation for communication performance of the adjacent node, determine a transmission route between the collection node and the first node, based on the received data, and
upon receiving data including an evaluation value for communication performance of the storage node, determine a transmission route between the collection node and the storage node, based on the received data.
2. The parallel processing apparatus of claim 1 , wherein
the plurality of nodes includes two or more nodes each including a non-volatile storage device;
the processor of the management node is configured to:
when a failure occurs in the first node, determine the storage node from among the two or more nodes, based on power consumption and/or memory usage of each of the two or more nodes, and
transmit data including the first instruction to the collection node; and
the processor of the collection node is configured to, upon receiving the first instruction, transmit data including the second instruction in the first direction and the second direction, respectively.
3. The parallel processing apparatus of claim 1 , wherein
the processor of the second node is configured to:
upon receiving data that is transmitted in the first direction and includes the second instruction, transmit, in the first direction, data including an evaluation value obtained by adding an evaluation value for communication performance of the second node to an evaluation value for communication performance of another node included in the received data, and
upon receiving data that is transmitted in the second direction and includes the second instruction, transmit, in the second direction, data including an evaluation value obtained by adding an evaluation value for communication performance of the second node to an evaluation value for communication performance of another node included in the received data.
4. The parallel processing apparatus of claim 1 , wherein
the processor of each of the adjacent node and the storage node is configured to:
upon receiving data that is transmitted in the first direction or in the second direction and includes the second instruction, transmit data including an evaluation value obtained by adding an evaluation value for communication performance of the each of the adjacent node and the storage node and an evaluation value for communication performance of another node included in the received data, to the collection node.
5. The parallel processing apparatus of claim 1 , wherein
the second node is configured to:
upon receiving plural pieces of data that are transmitted in the first direction and include the second instruction, specify a piece of data among the plural pieces of data, based on an evaluation value for communication performance of another node included in each of the plural pieces of data, and transmit, in the first direction, data obtained by adding an evaluation value for communication performance of the second node to the specified piece of data, and
upon receiving plural pieces of data that are transmitted in the second direction and include the second instruction, specify a piece of data among the plural pieces of data, based on an evaluation value for communication performance of another node included in each of the plural pieces of data, and transmit, in the second direction, data obtained by adding an evaluation value for communication performance of the second node to the specified piece of data.
6. The parallel processing apparatus of claim 1 , wherein
the processor of each of the adjacent node and the storage node is configured to:
upon receiving plural pieces of data that are transmitted in the first direction or the second direction and include the second instruction, specify a piece of data among the plural pieces of data, based on an evaluation value for communication performance of another node included in each of the plural pieces of data, and
transmit data including an evaluation value for communication performance of another node included in the specified piece of data and an evaluation value for communication performance of the each of the adjacent node and the storage node, to the collection node.
7. The parallel processing apparatus of claim 1 , wherein
the processor of the collection node is configured to:
transmit a third instruction for relaying information stored in the memory of the first node, to each node on the determined transmission route between the collection node and the first node, and
transmit a fourth instruction for converting information obtained by the third instruction, relaying the converted information, and storing the converted information in the memory of the storage node, to each node on the determined transmission route between the collection node and the storage node.
8. The parallel processing apparatus of claim 1 , wherein
the processor of the collection node is configured to:
transmit, in the first direction, data including the second instruction and first identification information identifying the first direction, and
transmit, in the second direction, data including the second instruction and second identification information identifying the second direction;
the processor of the second node is configured to:
upon receiving data including the first identification information, transmit data obtained by adding an evaluation value for communication performance of the second node to the received data, in the first direction, and
upon receiving data including the second identification information, transmit data obtained by adding an evaluation value for communication performance of the second node to the received data, in the second direction;
the processor of each of the adjacent node and the storage node is configured to, upon receiving data including the first identification information or the second identification information, transmit data obtained by adding an evaluation value for communication performance of the each of the adjacent node and the storage node to the received data, to the collection node; and
the processor of the collection node is configured to:
upon receiving data including the first identification information, determine a transmission route between the collection node and the first node, based on the received data, and
upon receiving data including the second identification information, determine a transmission route between the collection node and the storage node, based on the received data.
9. A method performed by a parallel processing apparatus including a plurality of nodes each including a memory and a processor configured to relay data between the plurality of nodes, the method comprising:
causing a management node among the plurality of nodes, which manages the plurality of nodes, to:
when a failure occurs in a first node of the plurality of nodes, determine a collection node that is included in the plurality of nodes and collects first information stored in the memory of the first node, based on power consumption and/or memory usage of each of the plurality of nodes, and
transmit, to the determined collection node, data including a first instruction for causing a storage node in the plurality of nodes to store second information generated from the first information into the memory of the storage node;
causing the collection node to, upon receiving the first instruction, transmit data including a second instruction that instructs transmission of an evaluation value for communication performance of each node, in a first direction approaching the first node and in a second direction approaching the storage node, respectively;
causing a second node that is included in the plurality of nodes and is neither an adjacent node adjacent to the first node nor the storage node, to:
upon receiving data that is transmitted in the first direction and includes the second instruction, transmit data obtained by adding an evaluation value for communication performance of the second node to the received data, in the first direction, and
upon receiving data that is transmitted in the second direction and includes the second instruction, transmit data obtained by adding an evaluation value for communication performance of the second node to the received data, in the second direction;
causing each of the adjacent node and the storage node to, upon receiving data that is transmitted in the first direction or in the second direction and includes the second instruction, transmit data including an evaluation value for communication performance of another node that is included in the received data and an evaluation value for communication performance of the each of the adjacent node and the storage node, to the collection node; and
causing the collection node to:
upon receiving data Including an evaluation for communication performance of the adjacent node, determine a transmission route between the collection node and the first node, based on the received data, and
upon receiving data including an evaluation value for communication performance of the storage node, determine a transmission route between the collection node and the storage node, based on the received data.
10. A non-transitory, computer-readable recording medium having stored therein a program for causing a computer included in a parallel processing apparatus that includes a plurality of nodes each including a memory and a processor configured to relay data between the plurality of nodes, to execute a process comprising:
causing a management node among the plurality of nodes, which manages the plurality of nodes, to:
when a failure occurs in a first node of the plurality of nodes, determine a collection node that is included in the plurality of nodes and collects first information stored in the memory of the first node, based on power consumption and/or memory usage of each of the plurality of nodes, and
transmit, to the determined collection node, data including a first instruction for causing a storage node in the plurality of nodes to store second information generated from the first information into the memory of the storage node;
causing the collection node to, upon receiving the first instruction, transmit data including a second instruction that instructs transmission of an evaluation value for communication performance of each node, in a first direction approaching the first node and in a second direction approaching the storage node, respectively;
causing a second node that is included in the plurality of nodes and is neither an adjacent node adjacent to the first node nor the storage node, to:
upon receiving data that is transmitted in the first direction and includes the second instruction, transmit data obtained by adding an evaluation value for communication performance of the second node to the received data, in the first direction, and
upon receiving data that is transmitted in the second direction and includes the second instruction, transmit data obtained by adding an evaluation value for communication performance of the second node to the received data, in the second direction;
causing each of the adjacent node and the storage node to, upon receiving data that is transmitted in the first direction or in the second direction and includes the second instruction, transmit data including an evaluation value for communication performance of another node that is included in the received data and an evaluation value for communication performance of the each of the adjacent node and the storage node, to the collection node; and
causing the collection node to:
upon receiving data including an evaluation for communication performance of the adjacent node, determine a transmission route between the collection node and the first node, based on the received data, and
upon receiving data including an evaluation value for communication performance of the storage node, determine a transmission route between the collection node and the storage node, based on the received data.