DISTRIBUTED LEARNING SYSTEM, MODEL LEARNING APPARATUS, DISTRIBUTED LEARNING METHOD, MODEL LEARNING PROGRAM
A federated learning system includes a plurality of model learning apparatus. Each model learning apparatus is connected to any of the other model learning apparatus via a network. The model learning apparatus includes a mini-batch extraction unit, a model parameter update unit, a dual-variable calculation/transmission unit, a dual-variable reception unit, and a dual-variable setting unit. The model parameter update unit is configured to perform learning using a dual variable, a step size, a mini-batch of model training data, a constraint parameter, and a coefficient γ using a predetermined optimal value n and a predetermined hyperparameter α, thereby updating a model parameter. The dual-variable calculation/transmission unit is configured to calculate and transmit a dual variable using the model parameter updated by the model parameter update unit and a coefficient γ for each other model learning apparatus connected to the model learning apparatus.
1 . A model learning apparatus constituting a federated learning system including a plurality of model learning apparatus, each model learning apparatus being connected to any of the other model learning apparatus via a network, wherein
a model parameter, a dual variable, a step size, model training data, and a constraint parameter are set to predetermined initial values, and
the model learning apparatus comprising processing circuitry configured to:
extract a predetermined amount of data as a mini-batch from the model training data;
perform learning using a dual variable, a step size, a mini-batch of model training data, a constraint parameter, and a coefficient γ using a predetermined optimal value n and a predetermined hyperparameter α, thereby updating a model parameter;
calculate and transmit a dual variable using the model parameter updated by the model parameter update unit and a coefficient γ for each other model learning apparatus connected to the model learning apparatus;
receive a dual variable from the other model learning apparatus connected to the model learning apparatus; and
set the received dual variable as a dual variable to be used for the next learning.
2 . A model learning apparatus constituting a federated learning system including a plurality of model learning apparatus, each model learning apparatus being connected to any of the other model learning apparatus via a network, wherein
a model parameter, a dual variable, a step size, model training data, and a constraint parameter are set to predetermined initial values, and
the model learning apparatus comprising processing circuitry configured to:
generate noise;
extract a predetermined amount of data as a mini-batch from the model training data;
perform learning using a dual variable, a step size, a mini-batch of model training data, constraint parameters, a coefficient γ using a predetermined optimal value η and a predetermined hyperparameter α, thereby updating a model parameter;
calculate and transmit a dual variable with the noise added using the model parameter updated by the model parameter update unit and a coefficient γ for each other model learning apparatus connected to the model learning apparatus;
receive a dual variable from the other model learning apparatus connected to the model learning apparatus; and
set the received dual variable as a dual variable to be used for the next learning.
3 . (canceled)
4 . The model learning apparatus according to claim 1 , wherein
the model parameter w i r,k+1 is updated as follows:
w
i
r
,
k
+
1
=
arg
min
u
(
f
_
(
u
,
ξ
i
r
,
k
)
+
η
2
γ
∑
j
∈
N
A
i
|
j
(
u
)
-
z
i
|
j
r
,
k
2
)
provided that w i r,k+1 denotes a model parameter, r denotes the number of repetitions of learning in the entire federated learning system, k denotes the number of learning iterations within the model learning apparatus, i and j each are symbols indicating model learning apparatus, f − denotes a cost function or a function can replace the cost function, u denotes a model parameter before update, ξ i r,k denotes a mini-batch of model training data, a coefficient γ is 1+αη, N denotes the number of model learning apparatus constituting the federated learning system, A i|j denotes a constraint parameter, and z i|j r denotes a dual variable, and
the dual variable is calculated as follows:
y
i
|
j
←
1
γ
{
(
1
-
αη
)
z
i
|
j
r
,
k
-
2
A
i
|
j
(
w
i
r
,
k
+
1
)
}
provided that y i|j denotes a dual variable.
5 . (canceled)
6 . The model learning apparatus according to claim 2 , wherein
the model parameter w i r,k+1 is updated as follows:
w
i
r
,
k
+
1
=
arg
min
u
(
f
_
(
u
,
ξ
i
r
,
k
)
+
η
2
γ
∑
j
∈
N
A
i
|
j
(
u
)
-
z
i
|
j
r
,
k
2
)
provided that w i r,k+1 denotes a model parameter, r denotes the number of repetitions of learning in the entire federated learning system, k denotes the number of learning iterations within the model learning apparatus, i and j each are symbols indicating model learning apparatus, f − denotes a cost function or a function can replace the cost function, u denotes a model parameter before update, ξ i r,k denotes a mini-batch of model training data, a coefficient γ is 1+αη, N denotes the number of model learning apparatus constituting the federated learning system, A i|j denotes a constraint parameter, and z i|j r denotes a dual variable, and
the dual variable with noise added is calculated as follows:
y
i
|
j
←
1
γ
{
(
1
-
αη
)
z
i
|
j
r
,
k
-
2
A
i
|
j
(
w
i
r
,
k
+
1
+
n
i
)
}
provided that y i|j denotes a dual variable with noise added, and n i denotes noise generated by the noise generation unit.
7 . A federated learning method using a federated learning system including a plurality of model learning apparatus, each model learning apparatus being connected to any of the other model learning apparatus via a network, wherein
a model parameter, a dual variable, a step size, model training data, and a constraint parameter are set to predetermined initial values,
each model learning apparatus executes an inner loop process a first predetermined number of times, and
each model learning apparatus executes an outer loop process for executing the inner loop process a second predetermined number of times,
in the inner loop process, the federated learning method comprising:
performing learning using a dual variable, a step size, a mini-batch of model training data, a constraint parameter, and a coefficient γ using a predetermined optimal value η and a predetermined hyperparameter α, thereby updating a model parameter;
calculating and transmitting a dual variable using the model parameter updated in the model parameter update step and a coefficient γ for each other model learning apparatus connected to the model learning apparatus;
receiving a dual variable from the other model learning apparatus connected to the model learning apparatus; and
setting the received dual variable as a dual variable to be used for the next learning.
8 . A non-transitory computer-readable recording medium on which a program recorded thereon for causing a computer to function as the model learning apparatus according to claim 1 .
9 . A federated learning system including a plurality of the model learning apparatus according to claim 1 , each model learning apparatus being connected to any of the other model learning apparatus.
10 . A federated learning system including a plurality of the model learning apparatus according to claim 2 , each model learning apparatus being connected to any of the other model learning apparatus.
11 . A federated learning method using a federated learning system including a plurality of model learning apparatus, each model learning apparatus being connected to any of the other model learning apparatus via a network, wherein
a model parameter, a dual variable, a step size, model training data, and a constraint parameter are set to predetermined initial values,
each model learning apparatus executes an inner loop process a first predetermined number of times, and
each model learning apparatus executes an outer loop process for executing the inner loop process a second predetermined number of times,
in the inner loop process, the federated learning method comprising:
generating noise;
performing learning using a dual variable, a step size, a mini-batch of model training data, a constraint parameter, and a coefficient γ using a predetermined optimal value η and a predetermined hyperparameter α, thereby updating a model parameter;
calculating and transmitting a dual variable with the noise added using the model parameter updated in the model parameter update step and a coefficient γ for each other model learning apparatus connected to the model learning apparatus;
receiving a dual variable from the other model learning apparatus connected to the model learning apparatus; and
setting the received dual variable as a dual variable to be used for the next learning.
12 . A non-transitory computer-readable recording medium on which a program recorded thereon for causing a computer to function as the model learning apparatus according to claim 2 .
13 . A non-transitory computer-readable recording medium on which a program recorded thereon for causing a computer to function as the model learning apparatus according to claim 4 .
14 . A non-transitory computer-readable recording medium on which a program recorded thereon for causing a computer to function as the model learning apparatus according to claim 6 .