Neural network batch normalization optimization method and apparatus
View Patent ↗A neural network batch normalization optimization method includes: setting a first network layer in a neural network as a starting layer; sequentially obtaining initial bias values of different network layers backwards starting from the starting layer; calculating equivalent bias values of the different network layers; determining whether there is a target network layer, wherein a ratio of the equivalent bias value corresponding to a previous layer of a target network layer to the equivalent bias value corresponding to the target network layer is no less than a pre-set threshold value; and if the target network layer is present, setting the bias values of the different network layers between the starting layer and the previous layer of the target network layer to zero, and taking the equivalent bias value of the target network layer as a bias value of the target network layer.
1. A neural network batch normalization optimization method applied to a terminal, the terminal comprising a processor, an input device, an output device and a memory, each of the processor, the input device, the output device electrically connected with the memory, wherein computerized programs are stored in the memory and comprising program instructions, a plurality of different network layers is comprised in a neural network, and the program instructions are executed by the processor to perform the neural network batch normalization optimization method, the method comprising:
Setting, using the processor, a first network layer in the neural network as an initial layer;
starting from the initial layer, successively obtaining, using the processor, initial bias values of the plurality of different network layers from front to back;
processing, using the processor, the initial bias values of the plurality of different network layers separately by invoking a preset algorithm to obtain equivalent bias values of the plurality of different network layers;
determining, using the processor, whether there is a target network layer, and a ratio between an equivalent bias value corresponding to a previous layer of the target network layer and an equivalent bias value corresponding to the target network layer is not less than a preset threshold; and
if the target network layer existed, setting bias values of the plurality of different network layers between the initial layer and the previous layer of the target network layer to zero, and taking, using the processor, the equivalent bias value of the target network layer as a bias value of the target network layer; and wherein
the equivalent bias value of the first network layer is the initial bias value of the first network layer; the equivalent bias value of an N-th network layer that is behind an (N−1)-th network layer, is calculated that a sum is obtained by multiplying the (N−1)-th equivalent bias value of the (N−1)-th network layer by weights of the N-th network layer, and the initial bias value of the N-th network layer is added to obtain the equivalent bias value of the N-th network layer;
wherein N is an integer equal to or more than two.
2. The neural network batch normalization optimization method as claimed in claim 1 , wherein the method further comprises:
obtaining, using the processor, initial activation functions of a plurality of different activation layers formed between the initial layer and the target network layer;
obtaining, using the processor, equivalent bias values of the plurality of different activation layers formed between the initial layer and the target network layer;
according to the equivalent bias values of the plurality of different activation layers, separately processing, using the processor, the initial activation functions of the plurality of different activation layers to obtain activation functions of the plurality of different activation layers between the initial layer and the target network layer; and
performing, using the processor, activation procession on the plurality of different activation layers between the initial layer and the target network layer, respectively, according to the activation functions of the different activation layers.
3. The neural network batch normalization optimization method as claimed in claim 2 , wherein after performing activation procession on the plurality of different activation layers formed between the initial layer and the target network layer, respectively, according to the activation functions of the plurality of different activation layers, the method further comprises:
confirming, using the processor, there is an A-th network layer existed between the initial layer and the target network layer, and there is an initial padding value of the A-th network layer, wherein, A is an integer and is not less than 1;
obtaining, using the processor, an equivalent bias value of the A-th network layer;
according to the equivalent bias value of the A-th network layer, processing, using the processor, the initial padding value of the A-th network layer to obtain a padding value of the A-th network layer.
4. The neural network batch normalization optimization method as claimed in claim 1 , wherein before setting the first network layer in the neural network as the initial layer, the method further comprises:
obtaining, using the processor, P batch normalization layers in the neural network, wherein P is a positive integer;
confirming, using the processor, there are P′ batch normalization layers in the P batch normalization layers, wherein, a previous network layer of each of the P′ batch normalization layers is a convolutional layer/full connection layer, 1≤P′≤P; and
merging, using the processor, each of the P′ batch normalization layers with a previous network layer corresponding to each of the P′ batch normalization layers.
5. The neural network batch normalization optimization method as claimed in claim 4 , wherein the method further comprises:
when there are N target network layers, obtaining, using the processor, a minimum equivalent bias value in the equivalent bias values corresponding to the N target network layers, wherein N is an integer greater than 1;
determining, using the processor, the network layer corresponding to the minimum equivalent bias value as the target network layer;
or when there are N target network layers, determining, using the processor, the last network layer in the N target network layers as the target network layer.
6. A neural network batch normalization optimization apparatus applied to a terminal, the terminal comprising a processor, an input device, an output device and a memory, each of the processor, the input device, the output device electrically connected with the memory, wherein computerized programs are stored in the memory and comprising program instructions, a plurality of different network layers is comprised in a neural network, and the program instructions are executed by the processor, the apparatus comprising:
a start layer setup module performed by the processor and configured to set a first network layer in the neural network as an initial layer;
starting from the initial layer, an initial bias value acquisition module performed by the processor and configured to successively obtain initial bias values of the plurality of different network layers from front to back;
a data processing module performed by the processor and configured to process the initial bias values of the plurality of different network layers separately by invoking a preset algorithm to obtain equivalent bias values of the plurality of different network layers;
a judging module performed by the processor and configured to determine whether there is a target network layer, and a ratio between an equivalent bias value corresponding to a previous layer of the target network layer and an equivalent bias value corresponding to the target network layer is not less than a preset threshold; and
an optimization processing module configured to set bias values of the plurality of different network layers between the initial layer and the previous layer of the target network layer to zero, and take the equivalent bias value of the target network layer as a bias value of the target network layer; and wherein
the equivalent bias value of the first network layer is the initial bias value of the first network layer; the equivalent bias value of an N-th network layer that is behind an (N−1)-th network layer, is calculated that a sum is obtained by multiplying the (N−1)-th equivalent bias value of the (N−1)-th network layer by weights of the N-th network layer, and the initial bias value of the N-th network layer is added to obtain the equivalent bias value of the N-th network layer;
wherein N is an integer equal to or more than two.
7. The neural network batch normalization optimization apparatus as claimed in claim 6 , wherein the apparatus further comprises an activation function processing module performed by the processor and configured to:
obtain initial activation functions of a plurality of different activation layers formed between the initial layer and the target network layer;
obtain equivalent bias values of the plurality of different activation layers formed between the initial layer and the target network layer;
according to the equivalent bias values of the plurality of different activation layers, separately process the initial activation functions of the plurality of different activation layers to obtain activation functions of the plurality of different activation layers between the initial layer and the target network layer; and
perform activation procession on the plurality of different activation layers between the initial layer and the target network layer, respectively, according to the activation functions of the different activation layers.
8. The neural network batch normalization optimization apparatus as claimed in claim 7 , wherein the apparatus further comprises a padding processing module performed by the processor and configured to:
confirm there is an A-th network layer between the initial layer and the target network layer, and there is an initial padding value of the A-th network layer, wherein, A is an integer and is not less than 1;
obtain an equivalent bias value of the A-th network layer;
according to the equivalent bias value of the A-th network layer, process the initial padding value of the A-th network layer to obtain a padding value of the A-th network layer.
9. The neural network batch normalization optimization apparatus as claimed in claim 8 , wherein the apparatus further comprises a merge processing module performed by the processor and configured to:
obtain P batch normalization layers in the neural network, wherein P is a positive integer;
confirm there are P′ batch normalization layers in the P batch normalization layers, wherein, a previous network layer of each of the P′ batch normalization layers is a convolutional layer/full connection layer, 1≤P′≤P; and
merge each of the P′ batch normalization layers with a previous network layer corresponding to each of the P′ batch normalization layers, respectively.
10. The neural network batch normalization optimization apparatus as claimed in claim 9 , wherein the apparatus further comprises a target network layer determination module performed by the processor and configured to:
when there are N target network layers, obtain a minimum equivalent bias value in the equivalent bias values corresponding to the N target network layers, wherein N is an integer greater than 1;
determine the network layer corresponding to the minimum equivalent bias value as the target network layer;
or when there are N target network layers, determine the last network layer in the N target network layers as the target network layer.
11. The neural network batch normalization optimization method as claimed in claim 2 , wherein the activation procession is performed using the following activation function:
relu
(
x
)
=
{
x
❘
0
x
≥
0
;
x
<
0
;
optimizing the above activation function, the optimized activation function is as follows:
relu( x +bias_eq)=relu_new( x )+bias_eq;
relu
_
new
(
x
)
=
{
x
-
bias_eq
;
x
≥
-
bias_eq
;
x
<
-
bias_eq
;
wherein: x is an input value of the activation function of the different activation layers before being adjusted, bias_eq is equivalent biases of the different activation layers, and relu_new(x) is the modified activation function.
12. The neural network batch normalization optimization method as claimed in claim 4 , wherein for a Batch Norm layer in the neural network, if a k-th layer is the Batch Norm layer and comprises a plurality of outputs, an i-th output of the k-th Batch Norm layer is expressed as follows:
out
k
[
i
,
:
,
:
]
=
data
k
[
,
:
,
:
]
-
data_mean
k
[
i
]
data_var
k
[
i
]
+
ɛ
*
gamma
k
[
i
]
+
beta
k
[
i
]
;
data_mean
k
[
i
]
=
mean
(
data
k
[
i
,
:
,
:
]
)
;
data_var
k
[
i
]
=
var
(
data
k
[
i
,
:
,
:
]
)
;
(
1
)
wherein, data k [i, :, :] is input data of the k-th layer, gamma k [i] and beta k [i] are parameters of the i-th path of the k-th network layer obtained by training, data_var k [i] is a variance of the i-th path of the k-th network layer obtained by training, data_mean k [i] is a mean difference of the i-th path of the k-th network layer obtained by training, ε is a constant;
continuing to perform expansion and merge procession on the formula (1) to obtain that an expression of the i-th output of the k-th network layer is simplified as follows:
out
k
[
i
,
:
,
:
]
=
gamma
k
[
i
]
data_var
k
[
i
]
+
ɛ
*
data
k
[
,
:
,
:
]
-
data_mean
k
[
i
]
data_var
k
[
i
]
+
ɛ
*
gamma
k
[
i
]
+
beta
k
[
i
]
=
weight
k
[
i
]
*
data
k
[
i
,
:
,
:
]
+
bias
k
[
i
]
;
weight
k
[
i
]
=
gamma
k
[
i
]
data_var
k
[
i
]
+
ɛ
;
(
2
)
defining:
bias
k
[
i
]
=
-
data_mean
k
[
i
]
data_var
k
[
i
]
+
ɛ
*
gamma
k
[
i
]
+
beta
k
[
i
]
;
wherein, weight k [i] represents a weight of the i-th path of the k-th network layer, bias k [i] represents the bias value of the i-th path of the k-th network layer before being merged.