Granted Patent B2

US 12,266,347 · App. 18/055,553 · Granted Apr 1, 2025

Inventors: Anshuman Tripathi (Mountain View, CA); Han Lu (Santa Clara, CA); Hasim Sak (Santa Clara, CA)

Assignee: Google LLC

G10L15/063G06N3/08G06N20/00G10L15/04G10L15/16

Abstract

Claims (62)

1. A computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations comprising:

receiving a sequence of acoustic frames characterizing a speech conversation between two or more speakers;

encoding, by an audio encoder of a speech recognition model, the sequence of acoustic frames into corresponding audio embeddings;

for each corresponding audio embedding:

receiving a speaker embedding associated with the corresponding audio embedding;

identifying a respective speaker among the two or more speakers that is associated with the corresponding audio embedding based on the speaker embedding;

concatenating the corresponding audio embedding with the speaker embedding; and

generating, using a masking model, a masked embedding corresponding to the identified respective speaker based on the corresponding audio embedding concatenated with the speaker embedding; and

for each respective speaker among the two or more speakers, generating, by a dedicated speaker branch of the speech recognition model for the respective speaker that receives each masked embedding corresponding to the respective speaker, a respective transcription that transcribes a respective segment of the speech conversation spoken by the respective speaker,

wherein a training process trains the masking model and the each of the dedicated speaker branches by:

in a first stage, training a single recurrent neural network-transducer (RNN-T) model using training examples;

dividing the single RNN-T model into each of the dedicated speaker branches; and

in a second stage, training the masking model and fine-tuning each of the dedicated speaker branches using the training examples by applying a respective masking loss to a segment of each training example of the training examples where a speaker is not speaking to minimize an RNN-T loss for each dedicated speaker branch of the masking model.

2. The method of claim 1 , wherein the speech recognition model comprises a recurrent neural network-transducer (RNN-T) architecture.

3. The method of claim 2 , wherein the dedicated speaker branch of the speech recognition model for the respective speaker comprises:

a dedicated label encoder for the respective speaker configured to:

receive, as input, sequences of non-blank symbols output by a dedicated final softmax layer for the respective speaker; and

generate labels for the respective speaker; and

a dedicated joint network for the respective speaker configured to:

receive, as input, the labels for the respective speaker generated by the dedicated label encoder and each masked embedding corresponding to the respective speaker; and

generate, at each of a plurality of output steps, a probability distribution over possible speech recognition hypotheses for the respective speaker.

4. The method of claim 3 , wherein generating the respective transcription that transcribes the respective segment of the speech conversation spoken by the respective speaker is based on the probability distribution over possible speech recognition hypotheses generated at each of the plurality of output steps for the respective speaker.

5. The method of claim 1 , wherein the operations further comprise:

for each corresponding audio embedding, determining a fixed input assigned to the respective speaker associated with the corresponding audio embedding,

wherein identifying the respective speaker among the two or more speakers that is associated with the corresponding audio frame is based on the fixed input.

6. The method of claim 1 , wherein the operations further comprise generating a transcript of the speech conversation between the two or more speakers based on the respective transcription generated for each respective speaker among the two or more speakers.

7. The method of claim 1 , wherein the operations further comprise displaying the transcription on a display screen in communication with the data processing hardware.

8. The method of claim 1 , wherein:

wherein the sequence of acoustic frames characterizing the speech conversation are segmented from a monophonic signal captured by a user device; and

the data processing hardware resides on the user device.

9. An automated speech recognition (ASR) model comprising:

an audio encoder configured to:

receive, as input, a sequence of acoustic frames characterizing a speech conversation between two or more speakers; and

generate, at each of a plurality of output steps, an audio embedding corresponding to each acoustic frame in the sequence of acoustic frames;

a masking model configured to:

receive, as input, the audio embedding generated by the audio encoder at each of the plurality of output steps;

receive, as input, a speaker embedding associated with each corresponding audio embedding generated by the audio encoder at each of the plurality of output steps, the speaker embedding identifying a respective speaker among the two or more speakers that is associated with the corresponding audio embedding; and

generate, at each of the plurality of output steps, a respective masked embedding for the respective speaker among the two or more speakers that is associated with the audio embedding generated by the audio encoder at the corresponding output step, the respective masked embedding based on a concatenation of the speaker embedding that identifies the respective speaker among the two or more speakers that is associated with the corresponding audio embedding at the corresponding output step; and

for each respective speaker among the two or more speakers, a dedicated speaker branch configured to:

receive, as input, each respective masked embedding generated by the masking model for the respective speaker; and

generate a respective transcription that transcribes a respective segment of the speech conversation spoken by the respective speaker,

wherein a training process trains the masking model and the each of the dedicated speaker branches by:

in a first stage, training a single recurrent neural network-transducer (RNN-T) model using training examples;

dividing the single RNN-T model into each of the dedicated speaker branches; and

in a second stage, training the masking model and fine-tuning each of the dedicated speaker branches using the training examples by applying a respective masking loss to a segment of each training example of the training examples where a speaker is not speaking to minimize an RNN-T loss for each dedicated speaker branch of the masking model.

10. The ASR model of claim 9 , wherein the speech recognition model comprises a recurrent neural network-transducer (RNN-T) architecture.

11. The ASR model of claim 10 , wherein the dedicated speaker branch for each respective speaker comprises:

a dedicated label encoder for the respective speaker configured to:

receive, as input, sequences of non-blank symbols output by a dedicated final softmax layer for the respective speaker; and

generate labels for the respective speaker; and

a dedicated joint network for the respective speaker configured to:

receive, as input, the labels for the respective speaker generated by the dedicated label encoder and each respective masked embedding generated by the masking model for the respective speaker; and

generate, at each of a plurality of output steps, a probability distribution over possible speech recognition hypotheses for the respective speaker.

12. The ASR model of claim 11 , wherein the dedicated speaker branch for the respective speaker generates the respective transcription that transcribes the respective segment of the speech conversation spoken by the respective speaker based on the probability distribution over possible speech recognition hypotheses generated at each of the plurality of output steps for the respective speaker.

13. The ASR model of claim 9 , wherein:

the masking model is further configured to receive, for each corresponding audio embedding, a fixed input assigned to the respective speaker among the two or more speakers that is associated with the corresponding audio embedding; and

the masking model is configured to generate the respective masked embedding for the respective speaker based on the fixed input received for each corresponding audio embedding.

14. The ASR model of claim 9 , wherein a number of dedicated speaker branches is scalable based on a number of the two or more speakers speaking in the speech conversation characterized by the sequence of acoustic frames.

15. The ASR model of claim 9 , wherein:

the sequence of acoustic frames characterizing the speech conversation are segmented from a monophonic signal captured by a user device; and

the ASR model executes on the user device.

16. The ASR model of claim 9 , wherein the respective transcription generated by the dedicated speaker branch for each respective speaker among the two or more speakers is displayed on a screen of a user device.

Assignments (1)

Continuity (2)

References Cited (23)