Multitask distributed learning system and method based on lottery ticket neural network
View Patent ↗A distributed learning system and method comprises the steps of: obtaining an initial global weight broadcast and applying the initial global weight to a local model; performing a simulation test on a local model using test data obtained in advance according to a pre-designated task to be performed by the distributed learning device; when the local model to which the global weight is applied passes the simulation test according to a pre-designated test pass criterion, pruning in a pre-designated manner for a plurality of elements of the global weight; initializing values of unpruned residual elements in the global weight; locally training the local model to which the initialized weight is applied using learning data prepared in advance according to the task, and transmitting a local weight to the central server; and receiving the updated global weight for the next round according to the local weight in the central server.
1 . A distributed learning method of a distributed learning device, comprising the steps of:
obtaining an initial global weight broadcast and transmitted from a central server in an initial round of distributed learning that is repeatedly performed round by round and applying the initial global weight to a local model, which is an artificial neural network;
performing a simulation test on a local model to which a global weight obtained in a current round is applied using test data obtained in advance according to a pre-designated task to be performed by the distributed learning device;
when the local model to which the global weight obtained in the current round is applied passes the simulation test according to a pre-designated test pass criterion, pruning in a pre-designated manner for a plurality of elements of the global weight obtained in the current round;
initializing values of unpruned residual elements in the global weight obtained in the current round;
locally training the local model to which the global weight with the initialized values of unpruned residual elements is applied using learning data prepared in advance according to the task, and transmitting a local weight, which is a weight updated by a local training, to the central server; and
receiving an updated global weight for a next round according to the local weight in the central server,
wherein the distributed learning method further comprises the steps of:
when the local model to which the global weight obtained in the current round is applied does not pass the simulation test according to the pre-designated test pass criterion, changing a test pass criterion for the next round to be lowered according to a pre-designated manner; and
applying a local weight obtained in a previous round to the local model, and locally training the local model using learning data prepared in advance according to the task to obtain a local weight,
when passing the simulation test, determining whether the test pass criterion has been changed in the previous round; and
when it is determined that the test pass criterion has been changed, restoring the changed test pass criterion to a pre-designated initial test pass criterion.
2 . The distributed learning method according to claim 1 ,
wherein the step of pruning includes performing pruning by replacing a value of an element having a low value in the global weight with 0 according to a pre-designated pruning ratio.
3 . The distributed learning method according to claim 2 ,
wherein the step of pruning further includes increasing the pruning ratio in a pre-designated manner until the pruning ratio reaches a pre-designated target pruning ratio such that more elements can be pruned in the next round than the number of elements pruned in the current round.
4 . The distributed learning method according to claim 2 ,
wherein the step of initializing values of residual elements replaces a value of a residual element that is not pruned in the global weight with a value of a corresponding element of the initial global weight.
5 . The distributed learning method according to claim 1 ,
wherein the updated global weight is obtained by calculating an average value of elements corresponding to each other in a plurality of local weights transmitted from each of a plurality of distributed learning devices.
6 . The distributed learning method according to claim 1 ,
wherein the initial global weight has a plurality of elements having a value obtained randomly within a pre-designated range.
7 . The distributed learning method according to claim 1 ,
wherein the step of performing a simulation test is performed using the learning data as the test data.
8 . A distributed learning device for performing distributed learning including a processor, wherein the processor
obtains an initial global weight broadcast and transmitted from a central server in an initial round of distributed learning that is repeatedly performed round by round and applies the initial global weight to a local model, which is an artificial neural network,
performs a simulation test on a local model to which a global weight obtained in a current round is applied using test data obtained in advance according to a pre-designated task to be performed by the distributed learning device, when the local model to which the global weight obtained in the current round is applied passes the simulation test according to a pre-designated test pass criterion, prunes in a pre-designated manner for a plurality of elements of the global weight, initializes values of unpruned residual elements in the global weight obtained in the current round,
locally trains the local model to which the global weight with the initialized values of unpruned residual elements is applied using learning data prepared in advance according to the task, and transmits a local weight, which is a weight updated by a local training, to the central server, and
receives and obtains an updated global weight for a next round according to the local weight in the central server,
wherein the processor
when the local model to which the global weight is applied does not pass the simulation test according to the pre-designated test pass criterion, changes a test pass criterion for the next round to be lowered according to a pre-designated manner; and
applies a local weight obtained in a previous round to the local model, thereby locally training the local model,
when passing the simulation test, determines whether the test pass criterion has been changed in the previous round, and when it is determined that the test pass criterion has been changed, restores the changed test pass criterion to a pre-designated initial test pass criterion.
9 . The distributed learning device according to claim 8 ,
wherein the processor performs pruning by replacing a value of an element having a low value in the global weight with 0 according to a pre-designated pruning ratio.
10 . The distributed learning device according to claim 9 ,
wherein the processor increases the pruning ratio in a pre-designated manner until the pruning ratio reaches a pre-designated target pruning ratio such that more elements can be pruned in the next round than the number of elements pruned in the current round.
11 . The distributed learning device according to claim 9 ,
wherein the processor replaces a value of a residual element that is not pruned in the global weight with a value of a corresponding element of the initial global weight, thereby initializing values of residual elements.
12 . The distributed learning device according to claim 8 ,
wherein the updated global weight is obtained by calculating an average value of elements corresponding to each other in a plurality of local weights transmitted from each of a plurality of distributed learning devices.
13 . The distributed learning device according to claim 8 ,
wherein the initial global weight has a plurality of elements having a value obtained randomly within a pre-designated range.
14 . The distributed learning device according to claim 8 ,
wherein the processor performs the simulation test using the learning data as the test data.