Network system for running distributed environment applications and fault tolerance method using the same
A network system for running distributed environment applications is provided, including a main network part for interface between devices belonging to a plurality of device groups, and a sub-network part for interface between devices belonging to sub-device group, in which the sub-device group includes at least some of the devices belonging to each of different device groups.
1 . A network system comprising:
a first interface device configured to interface devices of at least one first device group of a plurality of device groups with devices of at least one second device group of the plurality of device groups, wherein a device group of the plurality of device groups comprises a plurality of standby devices;
a plurality of second interface devices, wherein each interface device of the plurality of second interface devices corresponds to a different sub-device group of a plurality of sub-device groups and is configured to interface at least one first device of a respective sub-device group with at least one second device of the respective sub-device group, wherein each sub-device group of the plurality of sub-device groups comprises at least one device selected from each device group of the plurality of device groups; and
a computing device configured to:
identify failure occurrence information associated with a failure of a first device belonging to a device group of the plurality of device groups;
terminate a first sub-process associated with the first device in a first sub-device group;
identify, based on a time indicated in the failure occurrence information, an operation graph required for failure recovery, the operation graph including operations to be executed from a time point associated with a latest checkpoint of the first device to a time point at which the failure occurs;
identify, based on the operation graph required for failure recovery, a second sub-process corresponding to operations affected by the failure associated with the first sub-process; and
cause a second device to execute the second sub-process using the operation graph required for failure recovery, wherein the second device is in an idle state and belongs to a standby device group.
2 . The network system according to claim 1 , further comprising:
a plurality of third interface devices of a first type, wherein each interface device of the plurality of third interface devices is coupled to devices of a respective device group of the plurality of device groups.
3 . The network system according to claim 1 , wherein each interface device of the plurality of second interface devices comprises an interface device, of a first type, coupled to devices of the respective sub-device group.
4 . The network system according to claim 1 , wherein the second device belongs to a standby device group comprising standby devices in idle states.
5 . The network system according to claim 1 , wherein the computing device is configured to cause the second device to execute the second sub-process by switching the second device from an idle state to an active state.
6 . The network system according to claim 1 , wherein the computing device is further configured to:
based on a determination that the first device is in a normal state after resolving the failure, register the first device in a standby device group.
7 . The network system according to claim 1 , wherein the first device and the second device are devices belonging to a sub-device group corresponding to a same sub-network part.
8 . The network system according to claim 1 , wherein the computing device is configured to cause the second device to execute the second sub-process further based on a latest checkpoint associated with the first device.
9 . The network system according to claim 1 , wherein the computing device is configured to cause the second device to execute the second sub-process by: restoring data associated with the first device using a latest checkpoint.
10 . A method performed by a network system, the method comprising:
identifying failure occurrence information associated with a failure of a first device belonging to a device group of a plurality of device groups;
terminating a first sub-process associated with the first device;
identifying, based on a time indicated in the failure occurrence information, an operation graph required for failure recovery, the operation graph including operations to be executed from a time point associated with a latest checkpoint of the first device to a time point at which the failure occurs;
identifying, based on the operation graph required for failure recovery, a second sub-process corresponding to operations affected by the failure associated with the first sub-process; and
causing a second device to execute the second sub-process using the operation graph required for failure recovery, wherein the second device is in an idle state and belongs to a standby device group, and
wherein the first device and the second device are devices belonging to a sub-device group corresponding to a same sub-network part.
11 . The method according to claim 10 , wherein the causing the second device to execute the second sub-process is further based on a latest checkpoint associated with the first device.
12 . The method according to claim 10 , wherein the causing the second device to execute the second sub-process comprises:
restoring data associated with the first device using a latest checkpoint.
13 . The method according to claim 10 , wherein the second sub-process corresponds to calculation of an error between training data and a prediction of labeled output as part of training of a deep learning model.
14 . The method according to claim 10 , wherein the failure occurrence information indicates a timeout.
15 . One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors of a computing device, cause the computing device to:
identify failure occurrence information associated with a failure of a first device belonging to a device group of a plurality of device groups;
terminate a first sub-process associated with the first device;
identify, based on a time indicated in the failure occurrence information, an operation graph required for failure recovery, the operation graph including operations to be executed from a time point associated with a latest checkpoint of the first device to a time point at which the failure occurs;
identify, based on the operation graph required for failure recovery, a second sub-process corresponding to operations affected by the failure associated with the first sub-process; and
cause a second device to execute the second sub-process using the operation graph required for failure recovery, wherein the second device is in an idle state and belongs to a standby device group, wherein the first device and the second device are devices belonging to a sub-device group corresponding to a same sub-network part.
16 . The one or more non-transitory computer-readable media of claim 15 , wherein the instructions, when executed by the one or more processors, cause the computing device to cause the second device to execute the second sub-process further based on a latest checkpoint associated with the first device.
17 . The one or more non-transitory computer-readable media of claim 15 , wherein the instructions, when executed by the one or more processors, cause the computing device to cause the second device to execute the second sub-process by causing the computing device to:
restore data associated with the first device using a latest checkpoint.
18 . The one or more non-transitory computer-readable media of claim 15 , wherein the failure occurrence information indicates a timeout.