Model gradient update method and device
The present application provides a model gradient update method and device, for use in improving the accuracy of model training. A central server repeatedly executes a gradient update process until a stop condition is satisfied. One gradient update process comprises: receiving first gradients respectively sent by multiple nodes, the first gradients being obtained by each node using sample data to train a model to be trained of the node one or more times; obtaining a second gradient one the basis of the multiple first gradients and the probability of each node in the present gradient update process, the probability of each node in the present gradient update process being determined by an Actor-Critic network one the basis of the probability of each node in the last gradient update process; and sending the second gradient to the multiple nodes, respectively.
1. A model gradient update method, applied to a central server, comprising:
performing, by the central server, gradient update processes repeatedly until a stopping condition is satisfied; wherein one of the gradient update processes comprises:
receiving a plurality of first gradients transmitted by a plurality of nodes respectively, wherein the first gradients are obtained by each of the plurality of nodes using a sample data to train a model to be trained in each of the plurality of nodes one or more times;
obtaining a second gradient based on the plurality of first gradients and a first probability of each of the plurality of nodes in a current gradient update process, wherein the first probability of each of the plurality of nodes is determined by an Actor-Critic network based on a second probability of each of the plurality of nodes in a previous gradient update process; and
transmitting the second gradient to the plurality of nodes respectively, wherein the plurality of nodes respectively update a weight of the model to be trained using the second gradient;
wherein the Actor-Critic network comprises an Actor network, at least one Critic network, and a reward function;
the reward function is configured to determine a reward value based on probabilities of the plurality of nodes determined in the previous gradient update process, and transmit the reward value to the at least one Critic network;
the at least one Critic network is configured to determine a target Q value and transmit the target Q value to the Actor network; and
the Actor network is configured to determine the first probability of each of the plurality of nodes in the current gradient update process based on the target Q value;
wherein the reward function satisfies:
r
=
g
B
A
;
wherein A represents a first accuracy rate, B represents a second accuracy rate, a value of g is greater than or equal to 1; the first accuracy rate is an accuracy rate of a trained model obtained by the central server and the plurality of nodes based on a federated average learning algorithm; the second accuracy rate is an average of third accuracy rates respectively transmitted by the plurality of nodes; and the third accuracy rate is obtained during a model training process same as a model training process in which the first gradient is obtained by the plurality of nodes using the sample data to train the model to be trained in the plurality of nodes.
2. The method of claim 1 , wherein the target Q value is a smallest Q value among Q values determined by a plurality of Critic networks.
3. The method of claim 1 , wherein in response to a value of B/A being greater than 1, the value of g is greater than 1;
in response to the value of B/A being smaller than or equal to 1, the value of g is equal to 1.
4. The method of claim 1 , wherein the Actor-Critic network comprises three Critic networks;
for each of the three Critic networks, a first Q value determined in the current gradient update process is based on a Q value gradient and a second Q value determined during the previous gradient update process;
wherein the Q value gradient is determined based on a first parameter, and the first parameter satisfies following formula:
J
=
𝔼
(
s
,
a
)
~
D
1
2
[
k
(
Q
θ
3
(
s
t
,
a
t
)
-
(
r
(
s
t
,
a
t
)
+
γ
V
θ
3
(
s
t
+
1
,
a
t
+
1
)
)
2
+
1
(
Q
θ
i
(
s
t
,
a
t
)
-
(
r
(
s
t
,
a
t
)
+
γ
V
θ
i
(
s
t
+
1
,
a
t
+
1
)
)
2
]
;
V
θ
i
(
s
t
+
1
,
a
t
+
1
)
=
𝔼
a
~
π
ϕ
[
(
Q
θ
i
(
s
t
,
a
t
)
-
α
t
ln
q
(
π
t
(
a
t
❘
"\[LeftBracketingBar]"
s
t
)
]
;
wherein J represents the first parameter; t represents a number of times of the current gradient update process; k>0, 1>0, k+1=1; θ 1 , θ 2 , and θ 3 respectively represent three Critic networks; θ i represents a network corresponding to a minimum value among latest Q values determined by the three Critic networks respectively represented by θ 1 , θ 2 and θ 3 ; s t represents a state during a t-th gradient update process; a t represents a third probability of the plurality of nodes during the t-th gradient update process; Q θ i (s t , a t ) represents a third Q value determined by the Critic network corresponding θ i under s t , a t condition during the t-th gradient update process; Q θ 3 (s t , a t ) represents a Q value output by the Critic network corresponding θ 3 under s t , a t condition during the t-th gradient update process; r(s t , a t ) represents the reward value under s t , a t condition during the t-th gradient update process; a value of γ is larger than 0; π t (a t |s t ) represents a fourth probability of a t under s t condition; q represents an index of entropy; In q represents entropy; and a value of α t is not equal to 0.
5. The method of claim 4 , wherein a used in the current gradient update process is determined based on a α gradient and a α used in the previous gradient update process, and the α gradient satisfies following formula:
J
(
α
)
=
a
~
D
,
a
-
π
ϕ
[
-
α
t
-
1
(
ln
q
(
π
t
(
a
t
❘
s
t
)
+
H
)
]
;
wherein, J(α) represents the α gradient, α t-1 represents the α used in the previous gradient update process, and H represents an ideal minimum expected entropy.
6. The method of claim 4 , wherein the k and the l are determined based on a variance of the accuracy of the model before the gradient update processes in the plurality of nodes.
7. The method of claim 1 , wherein the first probability of the node output by the Actor network in the current gradient update process is determined based on a probability gradient and the second probability of the node output in the previous gradient update process, and the probability gradient satisfies following formula:
J
(
π
ϕ
)
=
(
s
,
a
)
~
D
[
α
t
ln
q
(
π
t
(
a
t
❘
s
t
)
-
(
Q
θ
i
(
s
t
,
a
t
)
]
;
wherein, J(π ϕ ) represents the probability gradient, t represents a number of the current gradient update process; θ 1 , θ 2 , and θ 3 respectively represent three Critic networks; θ i represents a network corresponding to a minimum value among latest Q values determined by the three Critic networks respectively represented by θ 1 , θ 2 and θ 3 ; s t represents a state during a t-th gradient update process; a t represents probability of the plurality of nodes during the t-th gradient update process; Q θ i (s t , a t ) represents a Q value determined by the Critic network corresponding θ i under s t , a t condition during the t-th gradient update process; π t (a t |s t ) represents probability of a t under s t condition; q represents an index of entropy; In q represents entropy; α t represents α used in the current gradient update process, and a value of α t is not equal to 0.
8. The method according to any one of claim 1 , wherein the central server and the plurality of nodes perform gradient updates based on a federated learning architecture.