Multi-talker speech recognizer
Various systems and methods for multi-talker speech separation and recognition are disclosed herein. In one example, a system includes a memory and a processor to process mixed speech audio received from a microphone. In an example, the processor can also separate the mixed speech audio using permutation invariant training, wherein a criterion of the permutation invariant training is defined on an utterance of the mixed speech audio. In an example, the processor can also generate a plurality of separated streams for submission to a speech decoder.
1. A system for multi-talker speech separation and recognition, comprising:
a processor; and
a memory device coupled to the processor, the memory device to store instructions that, when executed by the processor, cause the processor to:
process a mixed speech audio received from a microphone;
use permutation invariant training (PIT) to separate and trace speech streams in the mixed speech audio, wherein the PIT includes:
estimate, in a first stage, a first mask by applying a first deep learning model to the mixed speech audio; and
estimate, in a second stage, a second mask by applying a second deep learning model to the first estimated mask and the mixed speech audio;
generate a plurality of separated speech streams from the mixed speech audio for submission to a speech decoder, wherein the plurality of separated speech streams are generated based on the first estimated mask and the second estimated mask; and
decode, using the speech decoder, speech of a first separated speech stream of the plurality of separated speech streams.
2. The system of claim 1 , wherein:
the plurality of separated speech streams comprises the first separated speech stream and a second separated speech stream; and
the processor is to decode with the speech decoder the speech of the first separated speech stream based on the first separated speech stream and the second separated speech stream.
3. The system of claim 1 , wherein the processor is to trace the mixed speech audio using permutation invariant training to provide tags to a speech stream indicating a speaker identity.
4. The system of claim 1 , wherein the permutation invariant training applies a language model of prediction to separating the mixed speech audio.
5. The system of claim 1 , wherein the permutation invariant training applies an acoustic model of prediction to separating the mixed speech audio.
6. The system of claim 1 , wherein:
the microphone is a microphone array; and
the mixed speech audio is multi-channel mixed speech audio.
7. The system of claim 1 , wherein the processor is to apply a beam former to separate the mixed speech audio into different directional streams based on a direction of the mixed speech audio compared to a direction of the beam former.
8. The system of claim 1 , wherein the processor is to adapt the separation of the mixed speech audio by the permutation invariant training based on supervision of a true transcript.
9. The system of claim 1 , wherein the plurality of separated speech streams comprises senone posterior probabilities.
10. The system of claim 9 , wherein the senone posterior probabilities are optimized using a cross entropy objective function.
11. The system of claim 1 , wherein the permutation invariant training applies a model that separates and traces speech streams at the same time, where the separation and tracing uses a stacking system in which a first stage separation result is used to inform a second stage model that includes the mixed speech and the separated speech streams from the first stage.
12. A method for multi-talker speech separation and recognition, comprising:
processing a mixed speech audio received from a microphone;
using permutation invariant training (PIT) to separate and trace speech streams in the mixed speech audio, wherein the PIT includes:
estimate, in a first stage, a first mask by applying a first deep learning model to the mixed speech audio; and
estimate, in a second stage, a second mask by applying a second deep learning model to the first estimated mask and the mixed speech audio;
generating a plurality of separated speech streams from the mixed speech audio for submission to a speech decoder, wherein the plurality of separated speech streams are generated based on the first estimated mask and the second estimated mask; and
decoding, using the speech decoder, speech of a first separated speech stream of the plurality of separated speech streams.
13. The method of claim 12 , comprising:
separating the mixed speech audio into the first separated speech stream and a second separated speech stream; and
decoding with the speech decoder the speech of the first separated speech stream based on the first separated speech stream and the second separated speech stream.
14. The method of claim 12 , comprising tracing the mixed speech audio using permutation invariant training to provide tags to a speech stream indicating a speaker identity.
15. The method of claim 12 , wherein the permutation invariant training applies a language model of prediction to separating the mixed speech audio.
16. The method of claim 12 , wherein the permutation invariant training applies an acoustic model of prediction to separating the mixed speech audio.
17. The method of claim 12 , wherein:
the microphone is a microphone array; and
the mixed speech audio is multi-channel mixed speech audio.
18. The method of claim 12 , comprising applying a beam former to separate the mixed speech audio into different directional streams based on a direction of the mixed speech audio compared to a direction of the beam former.
19. The method of claim 12 , comprising adapting the separation of the mixed speech audio by the permutation invariant training based on supervision of a true transcript.
20. A storage device that stores computer-readable instructions that, in response to an execution by a processor, cause the processor to:
process a mixed speech audio received from a microphone;
use permutation invariant training (PIT) to separate and trace speech streams in the mixed speech audio, wherein the PIT includes:
estimate, in a first stage, a first mask by applying a first deep learning model to the mixed speech audio; and
estimate, in a second stage, a second mask by applying a second deep learning model to the first estimated mask and the mixed speech audio;
generate a plurality of separated speech streams from the mixed speech audio for submission to a speech decoder, wherein the plurality of separated speech streams are generated based on the first estimated mask and the second estimated mask; and
decode, using the speech decoder, speech of a first separated speech stream of the plurality of separated speech streams.
21. The storage device of claim 20 , wherein:
the plurality of separated speech streams comprises the first separated speech stream and a second separated speech stream; and
wherein the instructions instruct a processor to decode with the speech decoder the speech of the first separated speech stream based on the first separated speech stream and the second separated speech stream.