Neural network processing device
A neural network processing device includes: a convolution part which receives an input signal and a learning completion weight parameter, performs a convolution operation on the input signal and the learning completion weight parameter, and outputs a convolution signal that is a result value of the convolution operation; a batch adjustment which receives the convolution signal and a learning completion normalization parameter, and outputs an adjustment signal obtained by adjusting an output deviation of the convolution signal; and an activation part which converts the adjustment signal into an output signal based on an activation function.
1. A neural network processing device comprising:
a convolution part which receives an input signal and a learning completion weight parameter, performs a convolution operation on the input signal and the learning completion weight parameter, and outputs a convolution signal that is a result value of the convolution operation, the convolution part comprising:
a multiplier which multiplies the input signal and the learning completion weight parameter; and
a quantizer which quantizes an output of the multiplier and outputs the convolution signal;
a batch adjustment part which receives the convolution signal and a learning completion normalization parameter, and outputs an adjustment signal obtained by adjusting an output deviation of the convolution signal; and
an activation part which converts the adjustment signal into an output signal based on an activation function;
wherein the quantizer performs a quantization operation corresponding to:
z
q
=
{
M
z
>
M
round
(
z
×
2
t
-
1
÷
M
)
÷
2
t
-
1
×
M
-
M
<
z
≤
M
-
M
z
≤
M
,
where z is the output of the multiplier, t is a bit width of the convolution signal, M is a maximum value in a range −a to b, a and b are natural numbers other than 1, and z q is the convolution signal; and
wherein the quantizer includes a quantization setting to set a range of the convolution signal output from the quantizer such that it corresponds to a floating-point type range of an actual signal represented by a curve of a distribution of a signal output from the multiplier, thereby converting to a fixed-point type range that reduces a number of bits required for quantization.
2. The neural network processing device of claim 1 , wherein the activation part normalizes the output signal to a value between 0 and 1.
3. The neural network processing device of claim 1 , wherein the learning completion weight parameter w′ is calculated by the following Equation:
w
′
=
w
max
(
❘
"\[LeftBracketingBar]"
w
❘
"\[RightBracketingBar]"
)
,
where w is a learning parameter.
4. The neural network processing device of claim 1 , wherein the learning completion normalization parameter is normalized to a value between −1 and 1.
5. The neural network processing device of claim 1 , wherein the learning completion normalization parameter includes a first parameter and a second parameter, and
wherein the batch adjustment part includes:
a multiplier which multiplies the convolution signal and the first parameter;
a first quantizer which quantizes an output of the multiplier;
an adder which adds an output of the first quantizer and the second parameter; and
a second quantizer which quantizes an output of the adder.
6. The neural network processing device of claim 5 , wherein the first parameter is a value calculated by the following Equation:
A
′
=
max
(
❘
"\[LeftBracketingBar]"
w
❘
"\[RightBracketingBar]"
)
max
(
❘
"\[LeftBracketingBar]"
y
❘
"\[RightBracketingBar]"
)
A
,
where max(|w|) is a maximum value of an absolute value of a learning parameter w, max(|y|) is a maximum value of an absolute value of an output signal y,
A
=
γ
σ
B
,
γ
is a learning parameter, and σ B is a standard deviation of the input signal.
7. The neural network processing device of claim 5 , wherein the second parameter is a value calculated by the following Equation:
B
′
=
B
max
(
❘
"\[LeftBracketingBar]"
y
❘
"\[RightBracketingBar]"
)
,
where max(|y|) is a maximum value of an absolute value of an output signal y,
B
=
β
-
μ
B
γ
σ
B
,
each of γ and β is a learning parameter, μ B is a mean of the input signal, and σ B is a standard deviation of the input signal.
8. The neural network processing device of claim 1 , wherein the activation part converts the adjustment signal into the output signal to have a value between 0 and 1, based on the activation function.
9. The neural network processing device of claim 1 , wherein the input signal is an image signal.
10. A neural network processing device comprising:
an input layer;
an intermediate layer; and
an output layer, and
wherein the intermediate layer includes:
a convolution part which receives an input signal and a learning completion weight parameter from the input layer, performs a convolution operation on the input signal and the learning completion weight parameter, and outputs a convolution signal that is a result value of the convolution operation, the convolution part comprising:
a multiplier which multiplies the input signal and the learning completion weight parameter; and
a quantizer which quantizes an output of the multiplier and outputs the convolution signal;
a batch adjustment part which receives the convolution signal and a learning completion normalization parameter, and outputs an adjustment signal obtained by adjusting an output deviation of the convolution signal; and
an activation part which converts the adjustment signal into an output signal based on an activation function and outputs the output signal to the output layer;
wherein the quantizer performs a quantization operation corresponding to:
z
q
=
{
M
z
>
M
round
(
z
×
2
t
-
1
÷
M
)
÷
2
t
-
1
×
M
-
M
<
z
≤
M
-
M
z
≤
M
,
where z is the output of the multiplier, t is a bit width of the convolution signal, M is a maximum value in a range −a to b, a and b are natural numbers other than 1, and z q is the convolution signal; and
wherein the quantizer includes a quantization setting to set a range of the convolution signal output from the quantizer such that it corresponds to a floating-point type range of an actual signal represented by a curve of a distribution of a signal output from the multiplier, thereby converting to a fixed-point type range that reduces a number of bits required for quantization.
11. The neural network processing device of claim 10 , wherein the activation part normalizes the output signal to a value between 0 and 1.
12. The neural network processing device of claim 10 , wherein the learning completion weight parameter w′ is calculated by the following Equation:
w
′
=
w
max
(
❘
"\[LeftBracketingBar]"
w
❘
"\[RightBracketingBar]"
)
,
where w is a learning parameter.
13. The neural network processing device of claim 10 , wherein the learning completion normalization parameter is normalized to a value between −1 and 1.
14. The neural network processing device of claim 10 , wherein the learning completion normalization parameter includes a first parameter and a second parameter, and
wherein the batch adjustment part includes:
a multiplier which multiplies the convolution signal and the first parameter;
a first quantizer which quantizes an output of the multiplier;
an adder which adds an output of the first quantizer and the second parameter; and
a second quantizer which quantizes an output of the adder.
15. The neural network processing device of claim 14 , wherein the first parameter is a value calculated by the following Equation:
A
′
=
max
(
❘
"\[LeftBracketingBar]"
w
❘
"\[RightBracketingBar]"
)
max
(
❘
"\[LeftBracketingBar]"
y
❘
"\[RightBracketingBar]"
)
A
,
where max(|w|) is a maximum value of an absolute value of a learning parameter w, max(|y|) is a maximum value an absolute value of an output signal y,
A
=
γ
σ
B
,
is a learning parameter, and σ B is a standard deviation of the input signal.
16. The neural network processing device of claim 14 , wherein the second parameter is a value calculated by the following Equation:
B
′
=
B
max
(
❘
"\[LeftBracketingBar]"
y
❘
"\[RightBracketingBar]"
)
,
where max(|y|) is a maximum value of an absolute value of an output signal y,
B
=
β
-
μ
B
γ
σ
B
,
each of γ and β is a learning parameter, μ B is a mean of the input signal, and σ B is a standard deviation of the input signal.