Knowledge transfer in permutation invariant training for single-channel multi-talker speech recognition
View Patent ↗Provided are a speech recognition training processing method and an apparatus including the same. The speech recognition training processing method includes acquiring a multi-talker mixed speech signal from a plurality of speakers, performing permutation invariant training (PIT) model training on the multi-talker mixed speech signal based on knowledge from a single-talker speech recognition model and updating a multi-talker speech recognition model based on a result of the PIT model training.
1. A method of performing speech recognition training performed by at least one processor, the method comprising:
acquiring, by the at least one processor, a multi-talker mixed speech signal from a plurality of speakers;
performing, by the at least one processor, permutation invariant training (PIT) model training on the multi-talker mixed speech signal based on knowledge from a single-talker speech recognition model; and
updating, by the at least one processor, a multi-talker speech recognition model based on a result of the PIT model training.
2. The method of claim 1 , wherein the single-talker speech recognition model is a teacher model, and the multi-talker speech recognition model is a student model.
3. The method of claim 1 , wherein the PIT model training uses labels from the single-talker speech recognition model, the labels being posteriors from inputting a single-talker data corresponding to one or more of the plurality of speakers into the single-talker speech recognition model.
4. The method of claim 1 , further comprises:
performing PIT model training on a single talker feature corresponding to one or more of the plurality of speakers; and
transferring posteriors from the performing the PIT model training on the single talker feature as soft label input for the multi-talker speech recognition model.
5. The method of claim 1 , wherein the performing PIT model training comprises:
performing a bidirectional long-short term memory (BLSTM) operation on the multi-talker mixed speech signal by assigning soft labels that are posteriors from inputting a single-talker data corresponding to one or more of the plurality of speakers into the single-talker speech recognition model and generating a plurality of estimated output segments for multi-talker mixed speech signal; and
minimizing a minimal average cross entropy (CE) for utterances of all possible assignments between the plurality of estimated output segments and soft labels.
6. The method of claim 1 , wherein the minimal average cross entropy (CE) is determined based on equation (1) and (2):
J
=
1
S
min
s
′
∈
permu
(
S
)
∑
s
∑
t
∑
y
p
′
(
y
|
o
t
s
s
′
)
log
p
θ
s
(
y
|
o
t
)
(
1
)
p
′
(
y
|
o
t
s
s
′
)
=
λ
p
teacher
(
y
|
o
t
s
s
′
)
+
(
1
-
λ
)
p
t
,
s
s
′
ref
(
y
)
(
2
)
where
p
θ
s
(
y
|
o
t
)
is posterior of stream s estimated by the multi-talker speech recognition model,
p
t
,
s
s
′
ref
(
y
)
=
δ
(
y
,
l
t
s
s
′
)
is a reference distribution, and
p
teacher
(
y
|
o
t
s
s
′
)
is posterior estimated by the single-talker speech recognition model using original single-talker speech stream s′ s .
7. The method of claim 1 , wherein the PIT model training comprises:
performing a bidirectional long-short term memory recurrent neural networks (BLSTM-RNN) operation on the multi-talker mixed speech signal by assigning soft labels that are posteriors from the single-talker speech recognition model.
8. The method of claim 4 , wherein the soft label is generated using an ensemble of a plurality of different single-talker speech recognition models such that
p
teacher
(
y
|
o
t
s
s
′
)
=
∑
k
w
k
p
k
(
y
|
o
t
s
s
′
)
wherein
w
k
∈
[
0
,
1
]
and
∑
k
w
k
=
1
are interpolation weights and
p
k
(
y
t
s
s
′
|
o
t
)
is the posterior estimated by the different plurality of single-talker speech recognition models.
9. The method of claim 4 , wherein the soft label is generated using an ensemble of a plurality of different single-talker speech recognition models one by one, in ascending order of recognition performance on corresponding single-talker tasks.
10. A speech recognition training apparatus comprising:
at least one memory operable to store program code; and
at least one processor operable to read said program code and operate as instructed by said program code, said program code comprising:
acquiring code configured to acquire a multi-talker mixed speech signal from a plurality of speakers;
training code configured to perform permutation invariant training (PIT) model training on the multi-talker mixed speech signal based on knowledge from a single-talker speech recognition model; and
updating code configured to update a multi-talker speech recognition model based on a result of the PIT model training.
11. The speech recognition training apparatus of claim 10 , wherein the single-talker speech recognition model is a teacher model, and the multi-talker speech recognition model is a student model.
12. The speech recognition training apparatus of claim 10 , wherein the PIT model training uses labels from the single-talker speech recognition model, the labels being posteriors from inputting a single-talker data corresponding to one or more of the plurality of speakers into the single-talker speech recognition model.
13. The speech recognition training apparatus of claim 10 , wherein the training code is further configured to:
perform PIT model training on a single talker feature corresponding to one or more of the plurality of speakers; and
transfer posteriors from the performing the PIT model training on the single talker feature as soft label input for the multi-talker speech recognition model.
14. The speech recognition training apparatus of claim 10 , wherein the training code is further configured to:
perform a bidirectional long-short term memory (BLSTM) operation on the multi-talker mixed speech signal by assigning soft labels that are posteriors from inputting a single-talker data corresponding to one or more of the plurality of speakers into the single-talker speech recognition model and generating a plurality of estimated output segments for multi-talker mixed speech signal; and
minimize a minimal average cross entropy (CE) for utterances of all possible assignments between the plurality of estimated output segments and soft labels.
15. The speech recognition training apparatus of claim 10 , wherein the minimal average cross entropy (CE) is determined based on equation (1) and (2):
J
=
1
S
min
s
′
∈
permu
(
S
)
∑
s
∑
t
∑
y
p
′
(
y
|
o
t
s
s
′
)
log
p
θ
s
(
y
|
o
t
)
(
1
)
p
′
(
y
|
o
t
s
s
′
)
=
λ
p
teacher
(
y
|
o
t
s
s
′
)
+
(
1
-
λ
)
p
t
,
s
s
′
ref
(
y
)
(
2
)
where
p
θ
s
(
y
|
o
t
)
is posterior of stream s estimated by the multi-talker speech recognition model,
p
t
,
s
s
′
ref
(
y
)
=
δ
(
y
,
l
t
s
s
′
)
is a reference distribution, and
p
teacher
(
y
|
o
t
s
s
′
)
is posterior estimated by the single-talker speech recognition model using original single-talker speech stream s′ s .
16. The speech recognition training apparatus of claim 10 , wherein the PIT model training comprises:
performing a bidirectional long-short term memory recurrent neural networks (BLSTM-RNN) operation on the multi-talker mixed speech signal by assigning soft labels that are posteriors from the single-talker speech recognition model.
17. The speech recognition training apparatus of claim 13 , wherein the soft label is generated using an ensemble of a plurality of different single-talker speech recognition models such that
p
teacher
(
y
|
o
t
s
s
′
)
=
∑
k
w
k
p
k
(
y
|
o
t
s
s
′
)
wherein
w
k
∈
[
0
,
1
]
and
∑
k
w
k
=
1
are interpolation weights and
p
k
(
y
t
s
s
′
|
o
t
)
is posterior estimated by the different plurality of single-talker speech recognition models.
18. The speech recognition training apparatus of claim 13 , wherein the soft label is generated using an ensemble of a plurality of different single-talker speech recognition models one by one, in ascending order of recognition performance on corresponding single-talker tasks.
19. A non-transitory computer readable medium having stored thereon a program for a method of performing speech recognition, the method comprising:
acquiring a multi-talker mixed speech signal from a plurality of speakers;
performing permutation invariant training (PIT) model training on the multi-talker mixed speech signal based on knowledge from a single-talker speech recognition model; and
updating a multi-talker speech recognition model based on a result of the PIT model training.