DISTRIBUTED LEARNING METHOD, DISTRIBUTED LEARNING SYSTEM, SERVER, AND PROGRAM
A distributed learning technology that can accelerate convergence while ensuring security is provided. A learning step of generating, by an i-th client, a model →w i,j in a j-th cycle using learning data D i , an aggregation step of generating, by a server, a global model →w g,j in the j-th cycle from the models →w 1,j , . . . , →w N,j in the j-th cycle according to a predetermined formula, and an end condition determination step of, by the server, ending learning processing with the global model →w g,j in the j-th cycle as the global model →w g in a case where a predetermined end condition is satisfied, and otherwise, transmitting the global model →w g,j in the j-th cycle to the i-th client, and an initialization step of setting, by the i-th client, the global model →w g,j in the j-th cycle as an initial value of a model →w i,j+1 in a j+1-th cycle are included.
1 . A distributed learning method in which a distributed learning system including N (N is an integer of 2 or more) clients and a server generates a global model →w g , the distributed learning method comprising:
where A(→w 1 , . . . , →w N ) is a function that receives vectors →w 1 , . . . , →w N as inputs and outputs a vector, and i is an integer of 1 or more and N or less,
a learning step of generating, by an i-th client, a model →w i,j in a cycle of generating a j-th global model →w g,j (hereinafter, referred to as a j-th cycle) using learning data D i ;
an aggregation step of generating, by the server, a global model →w g,j in the j-th cycle from the models →w 1,j , . . . , →w N,j in the j-th cycle according to a following formula:
w
→
g
,
j
←
A
(
w
→
1
,
j
,
…
,
w
→
N
,
j
)
+
r
δ
(
j
)
[
Math
.
9
]
(where r is a random number and δ(j) is a monotonically decreasing function);
an end condition determination step of, by the server, ending learning processing with the global model →w g,j in the j-th cycle as the global model →w g in a case where a predetermined end condition is satisfied, and otherwise, transmitting the global model →w g,j in the j-th cycle to the i-th client; and
an initialization step of setting, by the i-th client, the global model →w g j in the j-th cycle as an initial value of a model →w i,j +1 in a j+1-th cycle.
2 . The distributed learning method according to claim 1 ,
wherein a function α(j) is a monotonically increasing function, and k is a predetermined constant, and
a function δ(j) is a function expressed as δ(j)=k/α(j).
3 . The distributed learning method according to claim 1 ,
wherein a function α(j) is a monotonically increasing function, and
a function δ(j) is a function expressed by a following formula.
δ
(
j
)
=
❘
"\[LeftBracketingBar]"
A
(
w
→
1
,
j
,
…
,
w
→
N
,
j
)
-
w
→
g
,
j
-
1
❘
"\[RightBracketingBar]"
α
(
j
)
[
Math
.
10
]
4 . The distributed learning method according to claim 1 ,
wherein a random number r is generated according to a Laplace distribution or a Gaussian distribution having an average of 0.
5 . A distributed learning system comprising N (N is an integer of 2 or more) clients and a server, the distributed learning system generating a global model →w g ,
wherein, where A(→w 1 , . . . , →w N ) is a function that receives vectors →w 1 , . . . , →w N as inputs and outputs a vector, and i is an integer of 1 or more and N or less,
an i-th client includes
a learning circuitry configured to generate a model →w i,j in a cycle of generating a j-th global model →w g,j (hereinafter, referred to as a j-th cycle) using learning data D i , and
an initialization circuitry configured to set a global model →w g,j in the j-th cycle as an initial value of a model →w i,j+1 in a j+1-th cycle, and
the server includes
an aggregation circuitry configured to generate the global model →w g,j in the j-th cycle from the models →w 1,j , . . . , →w N,j in the j-th cycle according to a following formula:
w
→
g
,
j
←
A
(
w
→
1
,
j
,
…
,
w
→
N
,
j
)
+
r
δ
(
j
)
[
Math
.
11
]
(where r is a random number and δ(j) is a monotonically decreasing function), and an end condition determination circuitry configured to end learning processing with the global model →w g,j in the j-th cycle as the global model →w g in a case where a predetermined end condition is satisfied, and otherwise, to transmit the global model →w g,j in the j-th cycle to the i-th client.
6 . A server included in a distributed learning system that generates a global model →w g , the server comprising:
where A(→w 1 , . . . , →w N ) is a function that receives vectors →w 1 , . . . , →w N as inputs and outputs a vector, i is an integer of 1 or more and N or less, and →w i,j is a generated model by an i-th client in a cycle of generating a j-th global model →w g,j (hereinafter, referred to as a j-th cycle) using learning data D i ,
an aggregation circuitry configured to generate a global model →w g,j in the j-th cycle from the models →w 1,j , . . . , →w N,j in the j-th cycle according to a following formula:
w
→
g
,
j
←
A
(
w
→
1
,
j
,
…
,
w
→
N
,
j
)
+
r
δ
(
j
)
[
Math
.
12
]
(where r is a random number and δ(j) is a monotonically decreasing function); and
an end condition determination circuitry configured to end learning processing with the global model →w g j in the j-th cycle as the global model →w g in a case where a predetermined end condition is satisfied, and otherwise, to transmit the global model →w g,j in the j-th cycle to the i-th client.
7 . A non-transitory recording medium recording a program for causing a computer to function as the server according to claim 6 .