Embedding Normalization Method and Electronic Device Using Same
A method of training a neural network model for predicting a click-through rate (CTR) of a user in an electronic device includes normalizing an embedding vector on the basis of a feature-wise linear transformation parameter, and inputting the normalized embedding vector into a neural network layer, wherein the feature-wise linear transformation parameter is defined such that the same value is applied to all elements of the embedding vector.
1 . A method of training a neural network model for predicting a click-through rate (CTR) of a user in an electronic device, the method comprising:
mapping a feature included in a feature vector to an embedding vector;
normalizing an embedding vector on the basis of a feature-wise linear transformation parameter; and
inputting the normalized embedding vector into a neural network layer,
wherein the feature-wise linear transformation parameter may be defined such that the same value is applied to all elements of the embedding vector during the normalizing.
2 . The method of claim 1 , wherein the normalizing include:
calculating a mean of the elements of the embedding vector;
calculating a variance of the elements of the embedding vector; and
normalizing the embedding vector on the basis of the mean, the variance, and the feature-wise linear transformation parameter.
3 . The method of claim 1 , wherein the feature-wise linear transformation parameter includes a scale parameter and a shift parameter.
4 . The method of claim 3 , wherein each of the scale parameter and the shift parameter is a vector having the same dimension as the embedding vector, and all elements thereof has the same value.
5 . The method of claim 3 , wherein each of the scale parameter and the shift parameter has a scalar value.
6 . The method of claim 1 , wherein the normalizing is an operation of performing calculation of Equation 1 below:
EN
(
e
x
)
=
γ
x
f
(
e
x
-
μ
x
σ
x
2
+
ϵ
)
+
β
x
f
,
μ
x
=
1
d
∑
k
(
e
x
)
k
,
σ
x
2
=
1
d
∑
k
(
(
e
x
)
k
-
μ
x
)
2
.
,
[
Equation
1
]
wherein, in Equation 1, e x is the embedding vector, d is a dimension of the embedding vector, μ x is the mean of all the elements of the embedding vector, σ x 2 is the variance of all the elements of the embedding vector, (e x ) k is a k th element of the embedding vector e x , and each of γ x f and β x f is the feature-wise linear transformation parameter.
7 . A computer program stored in a computer-readable recording medium in combination with hardware to execute the method of claim 1 .
8 . A neural network system for predicting a click through rate (CTR) of a user implemented by at least one electronic device, the neural network system comprising:
an embedding layer;
a normalization layer; and
a neural network layer model,
wherein the embedding layer maps a feature included in a feature vector to an embedding vector,
the normalization layer normalizes the embedding vector on the basis of a feature-wise linear transformation parameter,
the neural network layer performs a neural network operation on the basis of the normalized embedding vector, and
the feature-wise linear transformation parameter is defined such that the same value is applied to all elements of the embedding vector in the normalization process.
9 . The neural network system of claim 8 , wherein the normalization layer calculates a mean of the elements of the embedding vector, calculates a variance of the elements of the embedding vector, and normalizes the embedding vector on the basis of the mean, the variance, and the feature-wise linear transformation parameter.
10 . The neural network system of claim 8 , wherein the feature-wise linear transformation parameter includes a scale parameter and a shift parameter.
11 . The neural network system of claim 10 , wherein each of the scale parameter and the shift parameter is a vector in the same dimension as the embedding vector, and all elements thereof has the same value.
12 . The neural network system of claim 10 , wherein each of the scale parameter and the shift parameter has a scalar value.
13 . The neural network system of claim 8 , wherein the normalization layer performs calculation of Equation 2 below:
EN
(
e
x
)
=
γ
x
f
(
e
x
-
μ
x
σ
x
2
+
ϵ
)
+
β
x
f
,
μ
x
=
1
d
∑
k
(
e
x
)
k
,
σ
x
2
=
1
d
∑
k
(
(
e
x
)
k
-
μ
x
)
2
.
,
[
Equation
2
]
wherein, in Equation 2, e x is the embedding vector, d is a dimension of the embedding vector, μ x is the mean of all the elements of the embedding vector, σ x 2 is the variance of all the elements of the embedding vector, (e x ) k is a k th element of the embedding vector e x , and γ x f and β x f are the feature-wise linear transformation parameters.