Machine learning system, machine learning method, and program
View Patent ↗Machine learning techniques which allow machine learning to be performed even when a cost function is not a convex function are provided. A machine learning system includes a plurality of node portions which learn mapping that uses one common primal variable by machine learning based on their respective input data while sending and receiving information to and from each other. The machine learning is performed so as to minimize, instead of a cost function of a non-convex function originally corresponding to the machine learning, a proxy convex function serving as an upper bound on the cost function. The proxy convex function is represented by a formula of a first-order gradient of the cost function with respect to the primal variable or by a formula of a first-order gradient and a formula of a second-order gradient of the cost function with respect to the primal variable.
1. A machine learning method comprising:
learning a deep neural network by performing:
a step in which a plurality of node portions learn mapping that uses one common primal variable by machine learning based on their respective input data while sending and receiving information to and from each other, wherein the plurality of node portions is a plurality of distributed servers,
the plurality of node portions perform the machine learning so as to minimize, instead of a cost function of a non-convex function originally corresponding to the machine learning, a proxy convex function serving as an upper bound on the cost function,
the proxy convex function is represented by a formula of a first-order gradient of the cost function with respect to a primal variable,
V is a predetermined positive integer equal to or greater than 2; the plurality of node portions are node portions 1, . . . , V, and a set of node portions is ˜V={1, . . . , V}; B is a predetermined positive integer, with b=1, . . . , B and a set of positive integers equal to or less than B is ˜B={1, . . . , B}; a set of node portions connected with a node portion i is ˜N(i); tis an integer, with t=0, . . . , T−1, where T is a positive integer; the bth vector constituting a dual variable λ i|j at the node portion i with respect to a node portion j is λ i|j,b ; the dual variable λ i|j,b after the t+1th update is λ i|j,b (t+1) ; the bth vector constituting a dual auxiliary variable z i|j of the dual variable λ i|j is z i|j,b ; the dual auxiliary variable z i|j,b after the t+1th update is z i|j,b (t+1); the bth vector constituting a dual auxiliary variable y i|j of the dual variable is λ i|j is y i|j,b ; the dual auxiliary variable y i|j,b after the t+1th update is y i|j,b (t+1) ; the bth element of a primal variable w i of the node portion i is w i,b ; the dual auxiliary variable w i,b after the t+1th update is w i,b (t+1) ;
a cost function corresponding to the node portion i used in the machine learning is f i ; a first-order gradient of the cost function f i with respect to w i,b (t) is ∇f i (w i,b (t) ); I is an identity matrix;
O is a zero matrix; σ 1 is a predetermined positive number; η is a positive number; and a matrix A i|j at the node portion i with respect to a node portion j is defined by the formula below,
A
i
❘
j
=
{
I
(
i
>
j
,
j
∈
N
~
(
i
)
)
-
I
(
i
<
j
,
j
∈
N
~
(
i
)
)
O
(
otherwise
)
,
and
for t=0, . . . , T−1,
(a) a step in which the node portion i performs an update of the dual variable according to the formula below:
for
i
∈
V
˜
,
j
∈
N
˜
(
i
)
,
b
∈
B
˜
,
λ
i
❘
j
,
b
(
t
+
1
)
=
(
1
σ
1
I
+
η
A
i
❘
j
A
i
❘
j
T
)
-
1
[
A
i
❘
j
(
w
i
,
b
(
t
)
-
η
∇
f
i
(
w
i
,
b
(
t
)
)
)
+
1
σ
1
z
i
❘
j
,
b
(
t
)
]
where A T denotes the transpose matrix of a matrix A, and
(b) a step in which the node portion i performs an update of the primal variable according to the formula below:
for i∈˜V,b∈˜B, w i,b (t+1) =w i,b (t) −η(∀ f i ( w i,b (t) )+Σ A i|j T λ i|j,b (t+1) j∈N ( i )),
and
for some or all of t=0, . . . , T−1, the following are performed in addition to the step (a) and the step (b):
(c) a step in which, with i∈˜V, j∈˜N(i), and b∈˜B, at least one node portion i sends the dual auxiliary variable y i|j,b (t+1) to at least one node portion j, and
(d) a step in which, with i∈˜V, j∈˜N(i), and b∈˜B, the node portion i that has received a dual auxiliary variable y j∈i,b (t+1) sets z i|j,b (t+1)=y j|i,b (t+1) , and
performing machine learning wherein the machine learning is carried out even when the cost function is not a convex function, wherein
the primal variable is updated independently of the plurality of the node portions.