Granted Patent B2

US 12,211,488 · App. 18/571,553 · Granted Jan 28, 2025

Inventors: Ioannis Alexandros Assael (London, GB); Brendan Shillingford (London, GB); Joao Ferdinando Gomes de Freitas (London, GB)

Assignee: DeepMind Technologies Limited

G10L15/063G10L25/30

Abstract

Claims (56)

1. A method performed by one or more computers, the method comprising:

receiving a video that includes a plurality of video frames that depict a first speaker;

obtaining a first embedding characterizing the first speaker, comprising:

obtaining adaptation data for the first speaker, the adaptation data comprising one or more adaptation videos of the first speaker and a respective ground truth transcription for each of the one or more adaptation videos, and

determining the first embedding for the first speaker using the adaptation data by minimizing a loss function that measures, for each of the one or more adaptation videos, a respective error between the ground truth transcription of the adaptation video and a respective speech recognition output for the adaptation video; and

processing a first input comprising (i) the video and (ii) the first embedding using a visual speech recognition neural network having a plurality of parameters, wherein the visual speech recognition neural network is configured to process the video and the first embedding in accordance with trained values of the parameters to generate a speech recognition output that defines a sequence of one or more words being spoken by the first speaker in the video.

2. The method of claim 1 , wherein the visual speech recognition neural network is configured to:

generate, from the first embedding, an additional input channel; and

combine the additional channel with one or more of the frames in the video prior to processing the frames in the video to generate the speech recognition output.

3. The method of claim 1 , wherein the visual speech recognition neural network comprises a plurality of hidden layers, and wherein the neural network is configured to, for at least one of the hidden layers:

generate, from the first embedding, an additional hidden channel; and

combine the hidden channel and an output of the hidden layer prior to providing the output for processing by another hidden layer of the visual speech recognition neural network.

4. The method of claim 1 , further comprising:

obtaining pre-trained values for the model parameters that have been determined by training the visual speech recognition neural network on training data comprising training examples corresponding to a plurality of speakers that are different from the first speaker, wherein determining the first embedding comprises determining the first embedding using the pre-trained values and the adaptation data.

5. The method of claim 4 , wherein determining the first embedding comprises:

initializing the first embedding; and

updating the first embedding by repeatedly performing operations comprising:

processing each of one or more of the adaptation videos in the adaptation data and the first embedding using the visual speech recognition neural network in accordance with current values of the parameters to generate a respective speech recognition output for each of the one or more adaptation videos; and

updating the first embedding to minimize the loss function.

6. The method of claim 5 , wherein updating the first embedding to minimize the loss function comprises:

backpropagating gradients of the loss function through the visual speech recognition neural network to determine a gradient of the loss function with respect to the first embedding; and

updating the first embedding using the gradient of the loss function with respect to the first embedding.

7. The method of claim 5 , wherein the current values are equal to the pre-trained values and to the trained values and wherein the model parameters are fixed while determining the first embedding.

8. The method of claim 5 , wherein the operations further comprise:

updating the current values of the parameters of the visual speech recognition neural network based on gradients of the loss function with respect to the parameters of the visual speech recognition neural network, and wherein the trained values are equal to the current values after determining the first embedding vector.

9. The method of claim 1 , further comprising:

applying a decoder to the speech recognition output for the video to generate the sequence of one or more words being spoken by the first speaker in the video.

10. The method of claim 1 , wherein the speech recognition output comprises, for each of the video frames, a respective probability distribution over a vocabulary of text elements.

11. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising: receiving a video that includes a plurality of video frames that depict a first speaker; obtaining a first embedding characterizing the first speaker, comprising: obtaining adaptation data for the first speaker, the adaptation data comprising one or more adaptation videos of the first speaker and a respective ground truth transcription for each of the one or more adaptation videos, and determining the first embedding for the first speaker using the adaptation data by minimizing a loss function that measures, for each of the one or more adaptation videos, a respective error between the ground truth transcription of the adaptation video and a respective speech recognition output for the adaptation video; and processing a first input comprising (i) the video and (ii) the first embedding using a visual speech recognition neural network having a plurality of parameters, wherein the visual speech recognition neural network is configured to process the video and the first embedding in accordance with trained values of the parameters to generate a speech recognition output that defines a sequence of one or more words being spoken by the first speaker in the video.

12. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

receiving a video that includes a plurality of video frames that depict a first speaker;

obtaining a first embedding characterizing the first speaker, comprising:

obtaining adaptation data for the first speaker, the adaptation data comprising one or more adaptation videos of the first speaker and a respective ground truth transcription for each of the one or more adaptation videos, and

determining the first embedding for the first speaker using the adaptation data by minimizing a loss function that measures, for each of the one or more adaptation videos, a respective error between the ground truth transcription of the adaptation video and a respective speech recognition output for the adaptation video; and

processing a first input comprising (i) the video and (ii) the first embedding using a visual speech recognition neural network having a plurality of parameters, wherein the visual speech recognition neural network is configured to process the video and the first embedding in accordance with trained values of the parameters to generate a speech recognition output that defines a sequence of one or more words being spoken by the first speaker in the video.

13. The system of claim 12 , wherein the visual speech recognition neural network is configured to:

generate, from the first embedding, an additional input channel; and

combine the additional channel with one or more of the frames in the video prior to processing the frames in the video to generate the speech recognition output.

14. The system of claim 12 , wherein the visual speech recognition neural network comprises a plurality of hidden layers, and wherein the neural network is configured to, for at least one of the hidden layers:

generate, from the first embedding, an additional hidden channel; and

combine the hidden channel and an output of the hidden layer prior to providing the output for processing by another hidden layer of the visual speech recognition neural network.

15. The system of claim 12 , the operations further comprising:

obtaining pre-trained values for the model parameters that have been determined by training the visual speech recognition neural network on training data comprising training examples corresponding to a plurality of speakers that are different from the first speaker, wherein determining the first embedding comprises determining the first embedding using the pre-trained values and the adaptation data.

16. The system of claim 15 , wherein determining the first embedding comprises:

initializing the first embedding; and

updating the first embedding by repeatedly performing updating operations comprising:

processing each of one or more of the adaptation videos in the adaptation data and the first embedding using the visual speech recognition neural network in accordance with current values of the parameters to generate a respective speech recognition output for each of the one or more adaptation videos; and

updating the first embedding to minimize the loss function.

17. The system of claim 16 , wherein updating the first embedding to minimize the loss function comprises:

backpropagating gradients of the loss function through the visual speech recognition neural network to determine a gradient of the loss function with respect to the first embedding; and

updating the first embedding using the gradient of the loss function with respect to the first embedding.

18. The system of claim 16 , wherein the current values are equal to the pre-trained values and to the trained values and wherein the model parameters are fixed while determining the first embedding.

19. The system of claim 16 , wherein the updating operations further comprise:

updating the current values of the parameters of the visual speech recognition neural network based on gradients of the loss function with respect to the parameters of the visual speech recognition neural network, and wherein the trained values are equal to the current values after determining the first embedding vector.

20. The system of claim 12 , the operations further comprising:

applying a decoder to the speech recognition output for the video to generate the sequence of one or more words being spoken by the first speaker in the video.

Assignments (2)

Priority Claims (1)

Continuity (1)

References Cited (37)