Large margin training for attention-based end-to-end speech recognition
A method of attention-based end-to-end (E2E) automatic speech recognition (ASR) training, includes performing cross-entropy training of a model, based on one or more input features of a speech signal, performing beam searching of the model of which the cross-entropy training is performed, to generate an n-best hypotheses list of output hypotheses, and determining a one-best hypothesis among the generated n-best hypotheses list. The method further includes determining a character-based gradient and a word-based gradient, based on the model of which the cross-entropy training is performed and a loss function in which a distance between a reference sequence and the determined one-best hypothesis is maximized, and performing backpropagation of the determined character-based gradient and the determined word-based gradient to the model, to update the model.
1. A method of attention-based end-to-end (E2E) automatic speech recognition (ASR) training, the method comprising:
performing cross-entropy training of a model, based on one or more input features of a speech signal;
performing beam searching of the model of which the cross-entropy training is performed, to generate an n-best hypotheses list of output hypotheses;
determining a one-best hypothesis among the generated n-best hypotheses list;
determining a character-based gradient and a word-based gradient, based on the model of which the cross-entropy training is performed and a loss function in which a distance between a reference sequence and the determined one-best hypothesis is maximized; and
performing backpropagation of the determined character-based gradient and the determined word-based gradient to the model, to update the model.
2. The method of claim 1 , wherein the output hypotheses included in the generated n-best hypotheses list have highest posteriors among output sequences of the model of which the cross-entropy training is performed.
3. The method of claim 1 , wherein the loss function is represented as follows:
L
(
θ
)
=
∑
(
χ
,
s
)
∈
D
[
l
(
s
^
b
,
s
)
-
(
log
p
θ
(
s
|
χ
)
-
log
p
θ
(
s
^
b
|
χ
)
)
]
+
2
,
where (χ, s) denotes a sample in a training set D, χ denotes an input feature, s denotes the reference sample, ŝ b , denotes the one-best hypothesis, l(ŝ, s) denotes the distance between the reference sequence and the determined one-best hypothesis, and θ denotes model parameters.
4. The method of claim 1 , wherein the loss function is represented as follows:
L
(
θ
)
=
∑
(
χ
,
s
)
∈
D
[
l
(
s
^
b
,
s
)
-
(
score
θ
(
s
|
χ
)
-
score
θ
(
s
^
b
|
χ
)
)
]
+
2
,
where (χ, s) denotes a sample in a training set D, χ denotes an input feature, s denotes the reference sample, ŝ b , denotes the one-best hypothesis, l(ŝ, s) denotes the distance between the reference sequence and the determined one-best hypothesis, θ denotes model parameters, and score denotes a log posterior.
5. The method of claim 4 , wherein each of the character-based gradient and the word-based gradient is represented as follows:
∂
L
(
θ
)
∂
log
p
θ
(
s
i
)
=
-
2
γ
+
δ
(
s
i
)
∂
L
(
θ
)
∂
log
p
θ
(
s
^
b
,
i
)
=
2
γ
+
δ
(
s
^
b
,
i
)
,
where δ(⋅) denotes a Kronecker delta function, and γ+ denotes [l(ŝ b ,s)−(score θ (s)−score θ (ŝ b ))].
6. The method of claim 4 , wherein each of the character-based gradient and the word-based gradient is represented as follows:
∂
L
(
θ
)
∂
log
p
θ
(
s
i
)
=
-
2
γ
+
δ
(
s
i
)
δ
(
i
≥
w
)
∂
L
(
θ
)
∂
log
p
θ
(
s
^
b
,
i
)
=
2
γ
+
δ
(
s
^
b
,
i
)
δ
(
i
≥
w
)
,
where δ(⋅) denotes a Kronecker delta function, γ+ denotes [l(ŝ b ,s)−(score θ (s)−score θ (ŝ b ))], and ω denotes a first wrong token.
7. The method of claim 1 , further comprising, based on the backpropagation of the determined character-based gradient and the determined word-based gradient to the model being performed, performing again the beam searching of the model of which the cross-entropy training is performed, to regenerate the n-best hypotheses list.
8. An apparatus for attention-based end-to-end (E2E) automatic speech recognition (ASR) training, the apparatus comprising:
at least one memory configured to store program code; and
at least one processor configured to read the program code and operate as instructed by the program code, the program code including:
first performing code configured to cause the at least one processor to perform cross-entropy training of a model, based on one or more input features of a speech signal;
second performing code configured to cause the at least one processor to perform beam searching of the model of which the cross-entropy training is performed, to generate an n-best hypotheses list of output hypotheses;
first determining code configured to cause the at least one processor to determine a one-best hypothesis among the generated n-best hypotheses list;
second and third determining code configured to cause the at least one processor to determine a character-based gradient and a word-based gradient, based on the model of which the cross-entropy training is performed and a loss function in which a distance between a reference sequence and the determined one-best hypothesis is maximized; and
third performing code configured to cause the at least one processor to perform backpropagation of the determined character-based gradient and the determined word-based gradient to the model, to update the model.
9. The apparatus of claim 8 , wherein the output hypotheses included in the generated n-best hypotheses list have highest posteriors among output sequences of the model of which the cross-entropy training is performed.
10. The apparatus of claim 8 , wherein the loss function is represented as follows:
L
(
θ
)
=
∑
(
χ
,
s
)
∈
D
[
l
(
s
^
b
,
s
)
-
(
log
p
θ
(
s
|
χ
)
-
log
p
θ
(
s
^
b
|
χ
)
)
]
+
2
,
where (χ, s) denotes a sample in a training set D, χ denotes an input feature, s denotes the reference sample, ŝ b denotes the one-best hypothesis, l(ŝ, s) denotes the distance between the reference sequence and the determined one-best hypothesis, and θ denotes model parameters.
11. The apparatus of claim 8 , wherein the loss function is represented as follows:
L
(
θ
)
=
∑
(
χ
,
s
)
∈
D
[
l
(
s
^
b
,
s
)
-
(
score
θ
(
s
|
χ
)
-
score
θ
(
s
^
b
|
χ
)
)
]
+
2
,
where (χ, s) denotes a sample in a training set D, χ denotes an input feature, s denotes the reference sample, ŝ b , denotes the one-best hypothesis, l(ŝ, s) denotes the distance between the reference sequence and the determined one-best hypothesis, θ denotes model parameters, and score denotes a log posterior.
12. The apparatus of claim 11 , wherein each of the character-based gradient and the word-based gradient is represented as follows:
∂
L
(
θ
)
∂
log
p
θ
(
s
i
)
=
-
2
γ
+
δ
(
s
i
)
∂
L
(
θ
)
∂
log
p
θ
(
s
^
b
,
i
)
=
2
γ
+
δ
(
s
^
b
,
i
)
,
where δ(⋅) denotes a Kronecker delta function, and γ+ denotes [l(ŝ b ,s)−(score θ (s)−score θ (ŝ b ))].
13. The apparatus of claim 11 , wherein each of the character-based gradient and the word-based gradient is represented as follows:
∂
L
(
θ
)
∂
log
p
θ
(
s
i
)
=
-
2
γ
+
δ
(
s
i
)
δ
(
i
≥
w
)
∂
L
(
θ
)
∂
log
p
θ
(
s
^
b
,
i
)
=
2
γ
+
δ
(
s
^
b
,
i
)
δ
(
i
≥
w
)
,
where δ(⋅) denotes a Kronecker delta function, γ+ denotes [l(ŝ b ,s)−(score θ (ŝ)−score θ (ŝ b ))], and ω denotes a first wrong token.
14. The apparatus of claim 8 , wherein the second performing code is further configured to cause the at least one processor to, based on the backpropagation of the determined character-based gradient and the determined word-based gradient to the model being performed, perform again the beam searching of the model of which the cross-entropy training is performed, to regenerate the n-best hypotheses list.
15. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor of a device, cause the at least one processor to:
perform cross-entropy training of a model, based on one or more input features of a speech signal;
perform beam searching of the model of which the cross-entropy training is performed, to generate an n-best hypotheses list of output hypotheses;
determine a one-best hypothesis among the generated n-best hypotheses list;
determine a character-based gradient and a word-based gradient, based on the model of which the cross-entropy training is performed and a loss function in which a distance between a reference sequence and the determined one-best hypothesis is maximized; and
perform backpropagation of the determined character-based gradient and the determined word-based gradient to the model, to update the model.
16. The non-transitory computer-readable medium of claim 15 , wherein the output hypotheses included in the generated n-best hypotheses list have highest posteriors among output sequences of the model of which the cross-entropy training is performed.
17. The non-transitory computer-readable medium of claim 15 , wherein the loss function is represented as follows:
L
(
θ
)
=
∑
(
χ
,
s
)
∈
D
[
l
(
s
^
b
,
s
)
-
(
log
p
θ
(
s
|
χ
)
-
log
p
θ
(
s
^
b
|
χ
)
)
]
+
2
,
where (χ, s) denotes a sample in a training set D, χ denotes an input feature, s denotes the reference sample, ŝ b denotes the one-best hypothesis, l(ŝ, s) denotes the distance between the reference sequence and the determined one-best hypothesis, and θ denotes model parameters.
18. The non-transitory computer-readable medium of claim 15 , wherein the loss function is represented as follows:
L
(
θ
)
=
∑
(
χ
,
s
)
∈
D
[
l
(
s
^
b
,
s
)
-
(
score
θ
(
s
|
χ
)
-
score
θ
(
s
^
b
|
χ
)
)
]
+
2
,
where (χ, s) denotes a sample in a training set D, χ denotes an input feature, s denotes the reference sample, ŝ b denotes the one-best hypothesis, l(ŝ, s) denotes the distance between the reference sequence and the determined one-best hypothesis, θ denotes model parameters, and score denotes a log posterior.
19. The non-transitory computer-readable medium of claim 18 , wherein each of the character-based gradient and the word-based gradient is represented as follows:
∂
L
(
θ
)
∂
log
p
θ
(
s
i
)
=
-
2
γ
+
δ
(
s
i
)
∂
L
(
θ
)
∂
log
p
θ
(
s
^
b
,
i
)
=
2
γ
+
δ
(
s
^
b
,
i
)
,
where δ(⋅) denotes a Kronecker delta function, and γ+ denotes [l(ŝ b ,s)−(score θ (s)−score θ (ŝ b ))].
20. The non-transitory computer-readable medium of claim 18 , wherein each of the character-based gradient and the word-based gradient is represented as follows:
∂
L
(
θ
)
∂
log
p
θ
(
s
i
)
=
-
2
γ
+
δ
(
s
i
)
δ
(
i
≥
w
)
∂
L
(
θ
)
∂
log
p
θ
(
s
^
b
,
i
)
=
2
γ
+
δ
(
s
^
b
,
i
)
δ
(
i
≥
w
)
,
where δ(⋅) denotes a Kronecker delta function, γ+ denotes [l(ŝ b ,s)−(score θ (s)−score θ (ŝ b ))], and ω denotes a first wrong token.