Granted Patent B2

US 12,417,070 · App. 18/327,134 · Granted Sep 16, 2025

Inventors: Marcin Gorzel (Dublin, IE); Balineedu Adsumilli (Sunnyvale, CA)

Assignee: GOOGLE LLC

G06F3/165G06V20/41G10L25/51H04S5/005

Abstract

Claims (76)

1. A method of assigning spatial information to audio segments, comprising:

receiving first video frames that include a visual object;

receiving a first audio segment, wherein the first audio segment includes an auditory event associated with the visual object, and wherein the first audio segment is non-spatialized;

upon determining that second video frames do not include the visual object and that a first time difference between the first video frames and the second video frames are not longer than a certain time, using a motion vector of the visual object to assign a spatial location to the auditory event in at least one of the second video frames;

receiving a second audio segment, wherein the second audio segment includes the auditory event;

receiving third video frames, wherein the third video frames do not include the visual object;

upon determining that the third video frames do not include the visual object and that a second time difference between the first video frames and the third video frames are longer than the certain time, assigning the auditory event to a diffuse sound field; and

generating an audio output that includes spatial locations of the visual object to a listener.

2. The method of claim 1 , further comprising:

identifying visual objects including the visual object in the first video frames by steps comprising:

assigning respective visual labels to the visual objects in the first video frames;

identifying auditory events including the auditory event in the first audio segment by steps comprising:

separating the first audio segment into multiple tracks; and

assigning respective audio labels to the multiple tracks; and

identifying matches between some of the auditory events and some of the visual objects by automatically matching some of the respective audio labels to some of the respective visual labels.

3. The method of claim 1 , further comprising:

identifying an unmatched auditory event, wherein the unmatched auditory event is not matched to an identified visual object in the first video frames; and

presenting the unmatched auditory event in a user interface.

4. The method of claim 3 , further comprising:

receiving, from a user, an assignment of the unmatched auditory event to another visual object identified in the first video frames.

5. The method of claim 3 , further comprising:

receiving, from a user, an indication to assign the unmatched auditory event as diffuse sound.

6. The method of claim 3 , further comprising:

receiving, from a user, an indication to assign the unmatched auditory event as directional sound and a spatial direction for the unmatched auditory event.

7. The method of claim 1 , wherein the first audio segment is monophonic.

8. The method of claim 1 , further comprising:

using blind source separation to identify auditory events that include the auditory event in the first audio segment by decomposing the first audio segment into multiple tracks, each corresponding to a respective auditory event.

9. The method of claim 1 , further comprising:

using image recognition to identify visual objects including the visual object in the first video frames.

10. A system, comprising:

a memory; and

a processor, the processor configured to execute instructions stored in the memory to:

receive first video frames that include a visual object;

receive an audio segment, wherein the audio segment includes an auditory event associated with the visual object;

receive second video frames, wherein the second video frames do not include the visual object;

upon to determining that the second video frames do not include the visual object and that a first time difference between the first video frames and the second video frames are not longer than a certain time, use a motion vector of the visual object to assign a second spatial location to the auditory event in at least one of the second video frames;

receive a third audio segment, wherein the third audio segment includes the auditory event;

receive third video frames, wherein the third video frames do not include the visual object;

upon determining that the third video frames do not include the visual object and that a second time difference between the first video frames and the third video frames are longer than the certain time, assign the auditory event to a diffuse sound field; and

generate an audio output that includes spatial locations of the visual object to a listener.

11. The system of claim 10 , wherein the processor is further configured to execute instructions stored in the memory to:

identify visual objects including the visual object in the first video frames, wherein to identify the visual objects comprises to:

assign respective visual labels to the visual objects in the first video frames;

identify auditory events including the auditory event in the audio segment, wherein to identify the auditory events comprises to:

separate the audio segment into multiple tracks; and

assign respective audio labels to the multiple tracks; and

identify matches between some of the auditory events and some of the visual objects by automatically matching some of the respective audio labels to some of the respective visual labels.

12. The system of claim 10 , wherein the processor is further configured to execute instructions stored in the memory to:

identify an unmatched auditory event, wherein the unmatched auditory event is not matched to an identified visual object in the first video frames; and

present the unmatched auditory event in a user interface.

13. The system of claim 12 , wherein the processor is further configured to execute instructions stored in the memory to:

receive, from a user, an assignment of the unmatched auditory event to another visual object identified in the first video frames.

14. The system of claim 12 , wherein the processor is further configured to execute instructions stored in the memory to:

receive, from a user, an indication to assign the unmatched auditory event as diffuse sound.

15. The system of claim 12 , wherein the processor is further configured to execute instructions stored in the memory to:

receive, from a user, an indication to assign the unmatched auditory event as directional sound and a spatial direction for the unmatched auditory event.

16. The system of claim 10 , wherein the audio segment is monophonic.

17. The system of claim 10 , wherein the processor is further configured to execute instructions stored in the memory to:

use blind source separation to identify auditory events that include the auditory event in the audio segment by decomposing the audio segment into multiple tracks, each corresponding to a respective auditory event.

18. The system of claim 10 , wherein the processor is further configured to execute instructions stored in the memory to:

use image recognition to identify visual objects including the visual object in the first video frames.

19. A non-transitory computer-readable storage medium, comprising executable instructions that, when executed by a processor, perform operations for assigning spatial information to audio segments, the operations comprising:

receiving first video frames that include a visual object;

receiving a first audio segment, wherein the first audio segment includes an auditory event associated with the visual object, and wherein the first audio segment is non-spatialized;

upon determining that second video frames do not include the visual object and that a first time difference between the first video frames and the second video frames are not longer than a certain time, using a motion vector of the visual object to assign a spatial location to the auditory event in at least one of the second video frames;

receiving a second audio segment, wherein the second audio segment includes the auditory event;

receiving third video frames, wherein the third video frames do not include the visual object;

upon determining that the third video frames do not include the visual object and that a second time difference between the first video frames and the third video frames are longer than the certain time, assigning the auditory event to a diffuse sound field; and

generating an audio output that includes spatial locations of the visual object to a listener.

20. The non-transitory computer-readable storage medium of claim 19 , the operations further comprising:

identifying visual objects including the visual object in the first video frames by steps comprising:

assigning respective visual labels to the visual objects in the first video frames;

identifying auditory events including the auditory event in the first audio segment by steps comprising:

separating the first audio segment into multiple tracks; and

assigning respective audio labels to the multiple tracks; and

identifying matches between some of the auditory events and some of the visual objects by automatically matching some of the respective audio labels to some of the respective visual labels.

Assignments (1)

Continuity (2)

References Cited (36)