Method and device for student training networks with teacher networks
A method and device for training a neural network are disclosed. The method comprises: selecting, by a training device, a teacher network performing the same functions of a student network; and iteratively training the student network and obtaining a target network, through aligning distributions of features between a first middle layer and a second middle layer corresponding to the same training sample data, so as to transfer knowledge of features of a middle layer of the teacher network to the student network.
1. A method for training a neural network, the method comprising:
iteratively training a student network, through transferring knowledge of a teacher network to the student network, wherein iteratively training the student network comprises:
building an objective function of the student network, where the objective function includes a task specific loss function and a distance loss function, and the distance loss function is a function used to measure the distance between distributions of features of a first middle layer and distributions of features of a second middle layer based on same training sample data; and
iteratively training the student network by using the training sample data to obtain a target network, when an iteration number of training reaches a threshold or the objective function satisfies pre-defined convergence conditions,
wherein, the task specific loss function comprises a cross-entropy loss function and the objective function of the student network comprises:
= ( y true ,p S )+λ MMD 2 ( F T ,F S )
wherein, (y true ,p S ) refers to the cross-entropy loss function, MMD 2 (F T , F S ) refers to the distance loss function, λ refers to a weight of the distance loss function, y true refers to a ground-truth labels of the training sample data, and p S refers to an output classification probability of the student network, F S ∈ C S ×HW , F T ∈ C T ×HW , C T refers to a number of channels of F T , C S refers to a number of channels of F S , H is a larger one of heights of F T and F S , W is a larger one of widths of F T and F S .
2. The method of claim 1 , wherein the transferring knowledge of features of a teacher network to the student network comprises transferring knowledge of features of a middle layer of a teacher network through aligning distributions of features between a first middle layer and a second middle layer corresponding to the same training sample data.
3. The method of claim 2 , wherein the features of the first middle layer refer to a feature map F T output from a first specific network layer of the teacher network, and the features of the second middle layer refer to a feature map F S output from a second specific network layer of the student network.
4. The method of claim 1 , wherein the teacher network performs same functions of the student network.
5. The method of claim 1 , further comprising: interpolating F T and F S according to H and W.
6. The method according to claim 1 , wherein iteratively training the student network comprises:
obtaining the features of the first middle layer and the second middle layer corresponding to a training sample data in a current iteration;
calculating the objective function according to the training sample data in the current iteration and the corresponding features of the first middle layer and the second middle layer thereto, and adjusting weights of the student network according to the objective function; and
continuing to the training of next iteration based on the weight adjusted student network.
7. A device for training a neural network, the device comprises a training element that is configured to
iteratively training a student network, through transferring knowledge of a teacher network to the student network, wherein iteratively training the student network comprises:
building an objective function of the student network, where the objective function includes a task specific loss function and a distance loss function, and the distance loss function is a function used to measure the distance between distributions of features of a first middle layer and distributions of the features of a second middle layer based on same training sample data; and
iteratively training the student network by using the training sample data to obtain a target network, when an iteration number of training reaches a threshold or the objective function satisfies pre-defined convergence conditions,
wherein, the task specific loss function comprises a cross-entropy loss function and the objective function of the student network comprises:
= ( y true ,p S )+λ MMD 2 ( F T ,F S )
wherein, (y true ,p S ) refers to the cross-entropy loss function, MMD 2 (F T , F S ) refers to the distance loss function, λ refers to a weight of the distance loss function, y true refers to a ground-truth labels of the training sample data, and p S refers to an output classification probability of the student network, F S ∈ C S ×HW , F T ∈ C T ×HW , C T refers to a number of channels of F T , C S refers to a number of channels of F S , H is a larger one of heights of F T and F S , W is a larger one of widths of F T and F S .
8. The device according to claim 7 , wherein the teacher network performs same functions of the student network.
9. The device according to claim 7 , wherein the transferring knowledge of features of a teacher network to the student network comprises transferring knowledge of features of a middle layer of a teacher network through aligning distributions of features between a first middle layer and a second middle layer corresponding to the same training sample data.
10. The device according to claim 7 , wherein the features of the first middle layer refer to a feature map F T output from a first specific network layer of the teacher network, and the features of the second middle layer refer to a feature map F S output from a second specific network layer of the student network.
11. The device according to claim 7 , wherein the training element, configured to iteratively train the student network for multiple times, is configured to:
obtain the features of the first middle layer and the second middle layer corresponding to the training sample data in a current iteration;
calculate the objective function according to the training sample data in the current iteration and the corresponding features of the first middle layer and the second middle layer thereto, and adjust weights of the student network according to the objective function; and
continue to the training of next iteration based on the weight adjusted student network.
12. The device according to claim 10 , wherein the training element is specifically configured to: adjust weights of the student network according to the objective function with a gradient descent optimization function.
13. The device according to claim 7 , wherein MMD 2 (F T , F S ) in the objective function comprises:
1
C
T
2
∑
i
=
1
C
T
∑
i
′
=
1
C
T
k
(
f
T
i
•
f
T
i
•
2
,
f
T
i
′
•
f
T
i
′
•
2
)
+
1
C
S
2
∑
j
=
1
C
S
∑
j
′
=
1
C
S
k
(
f
S
i
•
f
S
j
•
2
,
f
S
i
′
•
f
S
j
′
•
2
)
-
2
C
T
C
S
∑
i
=
1
C
T
∑
j
=
1
C
S
k
(
f
T
i
•
f
T
i
•
2
,
f
S
i
•
f
S
j
•
2
)
wherein, k(·,·) refers to a preset kernel function, C T refers to number of channels of F T , C S refers to number of channels of F S , f T i· represents a vectorized feature map of an i-th channel of F T , f T i′· represents a vectorized feature map of an i′-th channel of F T , f S j· refers to a vectorized feature map of a j-th channel of F S , and f S j′· refers to a vectorized feature map of ta j′-th channel of F S .
14. The device according to claim 13 , wherein k (·,·) is a preset linear kernel function, a preset polynomial kernel function, or a preset Gaussian kernel function.
15. The device according to claim 13 , wherein the training element is further configured to interpolate F T and F S according to H and W.
16. A device for training a neural network, the device comprising a processor and at least one memory, the at least one memory storing at least one machine executable instruction, which is executed by the processor to iteratively training a student network, through transferring knowledge of a teacher network to the student network, wherein iteratively training the student network comprises:
building an objective function of the student network, where the objective function includes a task specific loss function and a distance loss function, and the distance loss function is a function used to measure the distance between distributions of features of a first middle layer and distributions of features of a second middle layer based on same training sample data; and
iteratively training the student network by using the training sample data to obtain a target network, when an iteration number of training reaches a threshold or the objective function satisfies pre-defined convergence conditions,
wherein, the task specific loss function comprises a cross-entropy loss function and the objective function of the student network comprises:
= ( y true ,p S )+λ MMD 2 ( F T ,F S )
wherein, (y true ,p S ) refers to the cross-entropy loss function, MMD 2 (F T , F S ) refers to the distance loss function, λ refers to a weight of the distance loss function, y true refers to a ground-truth labels of the training sample data, and p S refers to an output classification probability of the student network, F S ∈ C S ×HW , F T ∈ C T ×HW , C T refers to a number of channels of F T , C S refers to a number of channels of F S , H is a larger one of heights of F T and F S , W is a larger one of widths of F T and F S .
17. The device according to claim 16 , wherein the processor executes the at least one machine executable instruction to iteratively train the student network using the training sample data, the at least one machine executable instruction being executed by the processor to interpolate F T and F S according to H and W.
18. The device according to claim 16 , wherein the teacher network performs same functions of the student network.