Video-informed spatial audio expansion

1. A method of assigning spatial information to audio segments, comprising:

receiving a first audio segment, wherein the first audio segment is non-spatialized, and wherein the first audio segment is associated with first video frames;

identifying visual objects in the first video frames;

identifying auditory events in the first audio segment;

identifying a match between a visual object of the visual objects and an auditory event of the auditory events;

assigning a first spatial location to the auditory event based on a location of the visual object, wherein the first spatial location indicates coordinates of a point within a bounding box surrounding the visual object;

receiving a second audio segment, wherein the second audio segment includes the auditory event of the auditory events;

receiving second video frames, wherein the second video frames do not include the visual object;

in response to determining that the second video frames do not include the visual object and that a first time difference between the first video frames and the second video frames does not exceed a certain time, using a motion vector of the visual object to assign a second spatial location to the auditory event in at least one of the second video frames;

receiving a third audio segment, wherein the third audio segment includes the auditory event;

receiving third video frames, wherein the third video frames do not include the visual object;

in response to determining that the third video frames do not include the visual object and that a second time difference between the first video frames and the third video frames exceeds the certain time, assigning the auditory event to a diffuse sound field; and

generating an audio output by upmixing the auditory event and the first spatial location, the second spatial location, and the diffuse sound field into a spatial representation, wherein the audio output conveys spatial locations of the audio output to a listener.

2. The method of claim 1 , further comprising:

identifying an unmatched auditory event, wherein the unmatched auditory event is not matched to an identified visual object in the first video frames; and

presenting the unmatched auditory event in a user interface.

3. The method of claim 2 , further comprising:

receiving, from a user, an assignment of the unmatched auditory event to another visual object of the visual objects identified in the first video frames.

4. The method of claim 2 , further comprising:

receiving, from a user, an indication to assign the unmatched auditory event as diffuse sound.

5. The method of claim 2 , further comprising:

receiving, from a user, an indication to assign the unmatched auditory event as directional sound and a spatial direction for the unmatched auditory event.

6. The method of claim 1 , wherein the first video frames are frames of a spherical video.

7. The method of claim 1 , wherein the first audio segment is monophonic.

8. The method of claim 1 , wherein identifying the auditory events in the first audio segment comprises:

using blind source separation to identify the auditory events in the first audio segment by decomposing the first audio segment into multiple tracks, each corresponding to a respective auditory event.

9. The method of claim 1 , wherein identifying the visual objects in the first video frames comprises:

using image recognition to identify the visual objects in the first video frames.

10. The method of claim 1 , wherein the audio output is a multi-channel file, an Ambisonics file, in a stereophonic format, or a Binaural stereo file.

11. A system for assigning spatial information to audio segments, comprising:

a memory; and

a processor, the processor configured to execute instructions stored in the memory to:

receive a first audio segment,

wherein the first audio segment is non-spatialized, and

wherein the first audio segment is associated with first video frames;

identify visual objects in the first video frames;

identify auditory events in the first audio segment;

in response to identifying a match between a visual object of the visual objects and an auditory event of the auditory events, assign a first spatial information to the auditory event;

in response to not identifying the match, assign the auditory event to a diffuse sound field;

receive a second audio segment, wherein the second audio segment includes the auditory event of the auditory events;

receive second video frames, wherein the second video frames do not include the visual object;

use a motion vector of the visual object to assign a second spatial information to the auditory event in at least one of the second video frames;

receive a third audio segment, wherein the third audio segment includes the auditory event;

receive third video frames, wherein the third video frames do not include the visual object;

assign the auditory event to the diffuse sound field based on a condition related to a distance between the first video frames and the third video frames; and

generate an audio output by upmixing the auditory event and the first spatial information, the second spatial information, and the diffuse sound field into a spatial representation, wherein the audio output conveys the spatial information or the diffuse sound field to a listener of the audio output.

12. The system of claim 11 , wherein to assign the first spatial information to the auditory event comprises to:

assign a spatial location to the auditory event based on a location of the visual object.

13. The system of claim 12 , wherein the spatial location corresponds to a center of a bounding polygon of the visual object.

14. The system of claim 12 , wherein the instructions further include instructions to:

generate an audio file that includes the auditory event and the spatial location of the auditory event.

15. The system of claim 11 , wherein the instructions further include instructions to:

generate an audio file that includes the auditory event and diffuse sound information related to the auditory event.

16. A non-transitory computer-readable storage medium, comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising:

receiving a first audio segment,

wherein the first audio segment is non-spatialized, and

wherein the first audio segment is associated with first video frames;

identifying visual objects in the first video frames;

identifying auditory events in the first audio segment;

in response to identifying a match between a visual object of the visual objects and an auditory event of the auditory events, assigning a first spatial information to the auditory event;

in response to not identifying the match, assigning the auditory event to a diffuse sound field;

receiving a second audio segment, wherein the second audio segment includes the auditory event of the auditory events;

receiving second video frames, wherein the second video frames do not include the visual object;

using a motion vector of the visual object to assign a second spatial information to the auditory event in at least one of the second video frames;

receiving a third audio segment, wherein the third audio segment includes the auditory event;

receiving third video frames, wherein the third video frames do not include the visual object;

assigning the auditory event to the diffuse sound field based on a condition related to a distance between the first video frames and the third video frames; and

generating an audio output by upmixing the auditory event and the first spatial information, the second spatial information, and the diffuse sound field into a spatial representation, wherein the audio output conveys the spatial information or the diffuse sound field to a listener of the audio output.

17. The non-transitory computer-readable storage medium of claim 16 , wherein assigning the first spatial information to the auditory event comprises:

assigning a spatial location to the auditory event based on a location of the visual object.

18. The non-transitory computer-readable storage medium of claim 17 , wherein the spatial location corresponds to a center of a bounding polygon of the visual object.

19. The non-transitory computer-readable storage medium of claim 17 , wherein the operations further comprise:

generating an audio file that includes the auditory event and the spatial location of the auditory event.

20. The non-transitory computer-readable storage medium of claim 16 , wherein the operations further comprise:

generating an audio file that includes the auditory event and diffuse sound information related to the auditory event.