Token-wise training for attention based end-to-end speech recognition
A method of attention-based end-to-end (A-E2E) automatic speech recognition (ASR) training, includes performing cross-entropy training of a model, based on one or more input features of a speech signal, determining a posterior probability vector at a time of a first wrong token among one or more output tokens of the model of which the cross-entropy training is performed, and determining a loss of the first wrong token at the time, based on the determined posterior probability vector. The method further includes determining a total loss of a training set of the model of which the cross-entropy training is performed, based on the determined loss of the first wrong token, and updating the model of which the cross-entropy training is performed, based on the determined total loss of the training set.
1. A method of attention-based end-to-end (A-E2E) automatic speech recognition (ASR) training, the method comprising:
performing cross-entropy training of a model, based on one or more input features of a speech signal;
selecting a hypothesis with a longest correct prefix, from a plurality of hypotheses of the model of which the cross-entropy training is performed;
determining a posterior probability vector at a time of a first wrong token included in the selected hypotheses, among one or more output tokens of the model of which the cross-entropy training is performed;
determining a loss of the first wrong token at the time, based on the determined posterior probability vector;
determining a total loss of a training set of the model of which the cross-entropy training is performed, based on the determined loss of the first wrong token; and
updating the model of which the cross-entropy training is performed, based on the determined total loss of the training set.
2. The method of claim 1 , wherein the posterior probability vector at the time is determined as follows:
p t =Decoder( s t−1 ∈{r t−1 ,y t−1 },H enc ),
where t denotes the time, p t denotes the posterior probability vector at the time t, H enc denotes the one or more features that are encoded, y t−1 denotes an output token at a previous time t−1, r t−1 denotes a reference token at the previous time t−1, and s t−1 denotes a token randomly selected from {r t−1 ,y t−1 }.
3. The method of claim 1 , wherein the total loss of the training set is determined as follows:
L
(
θ
)
TWT
=
∑
(
y
,
r
)
∈
(
Y
,
R
)
l
θ
(
y
t
ω
,
r
t
ω
)
,
where L(θ) denotes the total loss of the training set, (Y,R) denotes hypothesis-reference pairs in the training set, t ω denotes the time, y t ω denotes the first wrong token at the time, r t ω denotes a reference token at the time, and l θ (y t ω , r t ω ) denotes the loss of the first wrong token.
4. The method of claim 3 , wherein the loss of the first wrong token is determined as follows:
l θ ( y t ω ,r t ω )=−log p t ω ,r t ω ,
where p t ω ,r t ω denotes a posterior probability of the reference token at the time.
5. The method of claim 3 , wherein the loss of the first wrong token is determined as follows:
l θ ( y t ω ,r t ω )=−log p t ω ,r t ω +log p t ω ,y t ω ,
where p t ω ,r t ω denotes a posterior probability of the reference token at the time, and p t ω ,y t ω denotes a posterior probability of the first wrong token at the time.
6. The method of claim 1 , wherein the total loss of the training set is determined as follows:
L
(
θ
)
TWTiB
=
∑
(
y
,
r
)
∈
(
Y
,
R
)
l
θ
(
y
t
jl
,
ω
jl
,
r
t
jl
,
ω
)
,
where L(θ) denotes the total loss of the training set, (Y,R) denotes hypothesis-reference pairs in the training set, t jl,ω denotes the time, y t jl,ω jl denotes the first wrong token at the time, r t jl,ω row denotes a reference token at the time, and l θ (y t jl,ω jl ,r t jl,ω ) denotes the loss of the first wrong token.
7. An apparatus for attention-based end-to-end (A-E2E) automatic speech recognition (ASR) training, the apparatus comprising:
at least one memory configured to store program code; and
at least one processor configured to read the program code and operate as instructed by the program code, the program code including:
performing code configured to cause the at least one processor to perform cross-entropy training of a model, based on one or more input features of a speech signal;
selecting code configured to cause the at least one processor to select a hypothesis with a longest correct prefix, from a plurality of hypotheses of the model of which the cross-entropy training is preformed;
first determining code configured to cause the at least one processor to determine a posterior probability vector at a time of a first wrong token included in the selected hypothesis, among one or more output tokens of the model of which the cross-entropy training is performed;
second determining code configured to cause the at least one processor to determine a loss of the first wrong token at the time, based on the determined posterior probability vector;
third determining code configured to cause the at least one processor to determine a total loss of a training set of the model of which the cross-entropy training is performed, based on the determined loss of the first wrong token; and
updating code configured to cause the at least one processor to update the model of which the cross-entropy training is performed, based on the determined total loss of the training set.
8. The apparatus of claim 7 , wherein the posterior probability vector at the time is determined as follows:
p t =Decoder( s t−1 ∈{r t−1 ,y t−1 },H enc ),
where t denotes the time, p t denotes the posterior probability vector at the time t, H enc denotes the one or more features that are encoded, y t−1 denotes an output token at a previous time t−1, r t−1 denotes a reference token at the previous time t−1, and s t−1 denotes a token randomly selected from {r t−1 ,y t−1 }.
9. The apparatus of claim 7 , wherein the total loss of the training set is determined as follows:
L
(
θ
)
TWT
=
∑
(
y
,
r
)
∈
(
Y
,
R
)
l
θ
(
y
t
ω
,
r
t
ω
)
,
where L(θ) denotes the total loss of the training set, (Y,R) denotes hypothesis-reference pairs in the training set, t ω denotes the time, y t ω denotes the first wrong token at the time, r t ω denotes a reference token at the time, and l θ (y t ω , r t ω ) denotes the loss of the first wrong token.
10. The apparatus of claim 9 , wherein the loss of the first wrong token is determined as follows:
l θ ( y t ω ,r t ω )=−log p t ω ,r t ω ,
where p t ω ,r t ω denotes a posterior probability of the reference token at the time.
11. The apparatus of claim 9 , wherein the loss of the first wrong token is determined as follows:
l θ ( y t ω ,r t ω )=−log p t ω ,r t ω +log p t ω ,y t ω ,
where p t ω ,r t ω denotes a posterior probability of the reference token at the time, and p t ω ,y t ω denotes a posterior probability of the first wrong token at the time.
12. The apparatus of claim 7 , wherein the total loss of the training set is determined as follows:
L
(
θ
)
TWTiB
=
∑
(
y
,
r
)
∈
(
Y
,
R
)
l
θ
(
y
t
jl
,
ω
jl
,
r
t
jl
,
ω
)
,
where L(θ) denotes the total loss of the training set, (Y,R) denotes hypothesis-reference pairs in the training set, t jl,ω denotes the time, y t jl,ω jl denotes the first wrong token at the time, r t jl,ω denotes a reference token at the time, and l θ (y t jl,ω jl , r t jl,ω ) denotes the loss of the first wrong token.
13. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor of a device, cause the at least one processor to:
perform cross-entropy training of a model, based on one or more input features of a speech signal;
select a hypotheses with a longest correct prefix, from a plurality of hypotheses of the model of which the cross-entropy training is performed;
determine a posterior probability vector at a time of a first wrong token included in the selected hypothesis, among one or more output tokens of the model of which the cross-entropy training is performed;
determine a loss of the first wrong token at the time, based on the determined posterior probability vector;
determine a total loss of a training set of the model of which the cross-entropy training is performed, based on the determined loss of the first wrong token; and
update the model of which the cross-entropy training is performed, based on the determined total loss of the training set.
14. The non-transitory computer-readable medium of claim 13 , wherein the posterior probability vector at the time is determined as follows:
p t =Decoder( s t−1 ∈{r t−1 ,y t−1 },H enc ),
where t denotes the time, p t denotes the posterior probability vector at the time t, H enc denotes the one or more features that are encoded, y t−1 denotes an output token at a previous time t−1, r t−1 denotes a reference token at the previous time t−1, and s t−1 denotes a token randomly selected from {r t−1 ,y t−1 }.
15. The non-transitory computer-readable medium of claim 13 , wherein the total loss of the training set is determined as follows:
L
(
θ
)
TWT
=
∑
(
y
,
r
)
∈
(
Y
,
R
)
l
θ
(
y
t
ω
,
r
t
ω
)
,
where L(θ) denotes the total loss of the training set, (Y,R) denotes hypothesis-reference pairs in the training set, t ω denotes the time, y t ω denotes the first wrong token at the time, r t ω denotes a reference token at the time, and l θ (y t ω , r t ω ) denotes the loss of the first wrong token.
16. The non-transitory computer-readable medium of claim 15 , wherein the loss of the first wrong token is determined as follows:
l θ ( y t ω ,r t ω )=−log p t ω ,r t ω ,
where p t ω ,r t ω denotes a posterior probability of the reference token at the time.
17. The non-transitory computer-readable medium of claim 15 , wherein the loss of the first wrong token is determined as follows:
l θ ( y t ω ,r t ω )=−log p t ω ,r t ω +log p t ω ,y t ω ,
where p t ω ,r t ω denotes a posterior probability of the reference token at the time, and p t ω ,y t ω denotes a posterior probability of the first wrong token at the time.