Self-supervised learning method based on permutation invariant cross entropy and electronic device thereof
View Patent ↗Provided is a self-supervised learning method based on permutation invariant cross entropy. A self-supervised learning method based on permutation invariant cross entropy performed by an electronic device includes: defining a cross entropy loss function for pre-training of an end-to-end speech recognition model; configuring non-transcription speech corpus data composed only of speech as input data of the cross entropy loss function; setting all permutations of classes included in the non-transcription speech corpus data as an output target and calculating cross entropy losses for each class; and determining a minimum cross entropy loss among the calculated cross entropy losses for each class as a final loss.
1 . A self-supervised learning method based on permutation invariant cross entropy performed by an electronic device, the self-supervised learning method comprising:
defining a cross entropy loss function for pre-training of an end-to-end speech recognition model;
configuring non-transcription speech corpus data composed only of speech as input data of the cross entropy loss function;
setting all permutations of classes included in the non-transcription speech corpus data as an output target and calculating cross entropy losses for each class;
determining a minimum cross entropy loss among the calculated cross entropy losses for each class as a final loss; and
training the end-to-end speech recognition model based on at least the determined final loss to use the end-to-end speech recognition model to recognize speech;
wherein the defining of the cross entropy loss function for pre-training of the end-to-end speech recognition model includes:
converting probabilities y n,c for each class for n-th speech data x n of the non-transcription speech corpus data input to the end-to-end speech recognition model into a one-hot vector; and
defining the cross entropy loss function by Equation 1 below according to the conversion into the one-hot vector:
l n =−log P ( y=y n |x n ) [Equation 1]
wherein P(y=y n |x n ) denotes a posterior probability of outputting a correct symbol for the input speech data x n .
2 . The self-supervised learning method of claim 1 , wherein, in the configuring of the non-transcription speech corpus data composed only of speech as the input data of the cross entropy loss function, the non-transcription speech corpus data is grouped into N speech data and configured in mini-batch units.
3 . The self-supervised learning method of claim 2 , wherein, in the defining of the cross entropy loss function for pre-training of the end-to-end speech recognition model, the cross entropy loss function is further defined by Equation 2 below and is generated as the non-transcription speech corpus data, the non-transcription speech corpus data being composed of mini-batch units including N speech data;
l
(
x
,
y
)
=
∑
n
=
1
N
-
log
P
(
y
=
y
n
❘
x
n
)
.
[
Equation
2
]
4 . The self-supervised learning method of claim 2 , wherein, in the configuring of the non-transcription speech corpus data composed only of speech as the input data of the cross entropy loss function, a mini-batch is composed of randomly sampled N pieces of voice data.
5 . The self-supervised learning method of claim 2 , wherein, in the configuring of the non-transcription speech corpus data composed only of speech as the input data of the cross entropy loss function, the non-transcription speech corpus data includes N classes having a same distribution.
6 . The self-supervised learning method of claim 2 , wherein, in the setting of all the permutations of the classes included in the non-transcription speech corpus data as the output target and calculating the cross entropy losses for each class, the cross entropy losses L π 0 , L π 1 for each class A and B for the non-transcription speech corpus data, the non-transcription speech corpus data being composed of first and second speech corpuses x 0 and x 1 , is calculated through Equation 3 below:
L π 0 =−log P ( y=A|x 0 )−log P ( y=B|x 1 )
L π 1 =−log P ( y=B|x 0 )−log P ( y=A|x 1 ). [Equation 3]
7 . The self-supervised learning method of claim 6 , wherein, in the determining of the minimum cross entropy loss among the calculated cross entropy losses for each class as the final loss, the minimum cross entropy loss among the cross entropy losses for each class calculated to correspond to a number of N possible permutations P according to the configuration of the mini-batch units including the N speech data is determined as the final loss according to Equation 4 below
L
=
min
π
p
∈
P
∑
n
=
1
N
-
log
P
(
y
=
y
π
p
(
n
)
❘
x
n
)
,
❘
"\[LeftBracketingBar]"
P
❘
"\[RightBracketingBar]"
=
N
!
[
Equation
4
]
wherein π p denotes one element constituting the N possible permutations P.
8 . An electronic device for self-supervised learning based on permutation invariant cross entropy, the electronic device comprising:
a memory configured to store a program for the self-supervised learning based on permutation invariant cross entropy; and
a processor configured to execute the program stored in the memory,
wherein, as the processor executes the program, the processor configures non-transcription speech corpus data composed only of speech as input data of a cross entropy loss function defined for pre-training of an end-to-end speech recognition model, and then sets all permutations of classes included in the non-transcription speech corpus data as an output target to calculate cross entropy losses for each class, determines a minimum cross entropy loss among the calculated cross entropy losses for each class as a final loss, and trains the end-to-end speech recognition model based on at least the determined final loss to use the end-to-end speech recognition model to recognize speech;
wherein the processor converts probabilities for each class for n-th speech data x n of the non-transcription speech corpus data input to the end-to-end speech recognition model into a one-hot vector to define a cross entropy loss function as shown in Equation 1 below:
l n =−log P ( y=y n |x n ) [Equation 1]
wherein P(y=y n |x n ) denotes a posterior probability of outputting a correct symbol for the input speech data x n .
9 . The electronic device of claim 8 , wherein the processor randomly samples the non-transcription speech corpus data, the non-transcription speech corpus data including N classes having a same distribution to generate a mini-batch including N speech data.
10 . The electronic device of claim 9 , wherein the processor further generates the cross entropy loss function defined by Equation 2 below as the non-transcription speech corpus data, the non-transcription speech corpus data being composed of mini-batch units including N speech data:
l
(
x
,
y
)
=
∑
n
=
1
N
-
log
P
(
y
=
y
n
❘
x
n
)
.
[
Equation
2
]
11 . The electronic device of claim 9 , wherein the processor calculates the cross entropy losses L π 0 , L π 1 for each class A and B for the non-transcription speech corpus data, the non-transcription speech corpus data being composed of first and second speech corpuses x 0 and x 1 , through Equation 3 below:
L π 0 =−log P ( y=A|x 0 )−log P ( y=B|x 1 )
L π 1 =−log P ( y=B|x 0 )−log P ( y=A|x 1 ). [Equation 3]
12 . The electronic device of claim 11 , wherein the processor determines the minimum cross entropy loss among the cross entropy losses for each class calculated to correspond to a number of N possible permutations P according to a configuration of a mini-batch unit including the N speech data as the final loss according to Equation 4 below:
L
=
min
π
p
∈
P
∑
n
=
1
N
-
log
P
(
y
=
y
π
p
(
n
)
❘
x
n
)
,
❘
"\[LeftBracketingBar]"
P
❘
"\[RightBracketingBar]"
=
N
!
[
Equation
4
]
wherein π p denotes one element constituting the N possible permutations P.