Granted Patent B2

US 12,273,697 · App. 18/042,258 · Granted Apr 8, 2025

Inventors: Aren Jansen (Mountain View, CA); Manoj Plakal (New York, NY); Dan Ellis (New York, NY); Shawn Hershey (Kirkland, WA); Richard Channing Moore, III (Brooklyn, NY)

Assignee: GOOGLE LLC

H04S7/301H04S2400/01

Abstract

Claims (34)

1. A computer-implemented method for upmixing audiovisual data, the computer-implemented method comprising:

obtaining, by a computing system comprising one or more computing devices, audiovisual data comprising input audio data and video data accompanying the input audio data, wherein each frame of the video data depicts only a portion of a larger scene, and wherein the input audio data has a first number of audio channels;

providing, by the computing system, the audiovisual data as input to a machine-learned audiovisual upmixing model, the audiovisual upmixing model comprising a sequence-to-sequence model configured to model a respective location of one or more audio sources within the larger scene over multiple frames of the video data; and

receiving, by the computing system, upmixed audio data from the audiovisual upmixing model, the upmixed audio data having a second number of audio channels, the second number of audio channels greater than the first number of audio channels.

2. The computer-implemented method of claim 1 , wherein the audiovisual upmixing model comprises an encoder-decoder model.

3. The computer-implemented method of claim 1 , wherein the audiovisual upmixing model comprises a transformer model.

4. The computer-implemented method of claim 1 , wherein the audiovisual upmixing model comprises an attention mechanism.

5. The computer-implemented method of claim 4 , wherein the attention mechanism comprises a plurality of context vectors and an alignment model.

6. The computer-implemented method of claim 1 , wherein the audiovisual upmixing model comprises a plurality of input streams, each of the plurality of input streams corresponding to a respective audio channel of the input audio data, and a plurality of output streams, each of the plurality of output streams corresponding to a respective audio channel of the upmixed audio data.

7. The computer-implemented method of claim 1 , wherein the video data comprises two-dimensional video data.

8. The computer-implemented method of claim 1 , wherein the input audio data comprises mono audio data, the mono audio data having a single audio channel.

9. The computer-implemented method of claim 1 , wherein the upmixed audio data comprises stereo audio data, the stereo audio data having a left audio channel and a right audio channel.

10. The computer-implemented method of claim 1 , wherein the input audio data comprises stereo audio data, the stereo audio data having a left audio channel and a right audio channel.

11. The computer-implemented method of claim 1 , wherein the upmixed audio data comprises surround sound audio data, the surround sound audio data having three or more audio channels.

12. The computer-implemented method of claim 1 , wherein training the machine-learned audiovisual upmixing model comprises:

obtaining, by the computing system, audiovisual training data comprising video training data and audio training data having the second number of audio channels;

downmixing, by the computing system, the audio training data to produce downmixed audio training data comprising the first number of audio channels;

providing, by the computing system, the video training data and corresponding downmixed audio training data to the audiovisual upmixing model;

obtaining, by the computing system, a predicted upmixed audio data output comprising the second number of audio channels from the audiovisual upmixing model;

determining, by the computing system, a difference between the predicted upmixed audio data and the audio training data; and

updating one or more parameters of the model based the difference.

13. A computing system configured for upmixing audiovisual data, the computing system comprising:

one or more processors; and

one or more memory devices storing computer-readable data comprising instructions that, when implemented, cause the one or more processors to perform operations, the operations comprising:

obtaining audiovisual data comprising input audio data and video data accompanying the input audio data, the input audio data having a first number of audio channels;

providing the audiovisual data as input to a machine-learned audiovisual upmixing model, the audiovisual upmixing model comprising a sequence-to-sequence model; and

receiving upmixed audio data from the audiovisual upmixing model, the upmixed audio data having a second number of audio channels, the second number of audio channels greater than the first number of audio channels.

14. The computing system of claim 13 , wherein the audiovisual upmixing model comprises an encoder-decoder model.

15. The computing system of claim 13 , wherein the audiovisual upmixing model comprises a transformer model.

16. The computing system of claim 13 , wherein the audiovisual upmixing model comprises an attention mechanism.

17. The computing system of claim 16 , wherein the attention mechanism comprises a plurality of context vectors and an alignment model.

18. The computing system of claim 13 , wherein the audiovisual upmixing model comprises a plurality of internal state vectors.

19. The computing system of claim 13 , wherein the audiovisual upmixing model comprises a plurality of input streams, each of the plurality of input streams corresponding to a respective audio channel of the input audio data, and a plurality of output streams, each of the plurality of output streams corresponding to a respective audio channel of the upmixed audio data.

20. The computing system of claim 13 , wherein the video data comprises two-dimensional video data.

Assignments (1)

Continuity (1)

References Cited (17)