Audio renderer based on audiovisual information
An audio renderer can have a machine learning model that jointly processes audio and visual information of an audiovisual recording. The audio renderer can generate output audio channels. Sounds captured in the audiovisual recording and present in the output audio channels are spatially mapped based on the joint processing of the audio and visual information by the machine learning model. Other aspects are described.
1. A method comprising:
receiving a visual feature associated with a video signal and an audio feature associated with a sound captured in a microphone signal;
receiving metadata comprising a target scene that includes a visual representation of the sound; and
determining, as output of a machine learning (ML) model using 1) the visual feature, 2) the audio feature, and 3) the target scene as input, one or more mapping parameters that, when applied to the microphone signal, generates one or more output audio channels that includes the sound of the microphone signal that is spatially mapped according to a location of the visual representation within the target scene and the mapping parameters are determined based on one or more correlations between the sound and the visual feature.
2. The method of claim 1 , wherein the one or more mapping parameters includes a direction-of-arrival (DoA) estimation of the sound.
3. The method of claim 1 , wherein the one or more mapping parameters includes the location of the visual representation of the sound within the target scene.
4. The method of claim 1 , wherein the target scene is visually the same as a record scene represented by the video signal.
5. The method of claim 1 , wherein the visual representation is a virtual object associated with a recorded object that is a sound source of the sound within a recorded scene of the video signal, wherein the location of the virtual object within the target scene is different than a location of the recorded object within the recorded scene.
6. The method of claim 1 , wherein the one or more output audio channels are associated with a target output audio format that is one of: a binaural audio format comprising a left audio channel and a right audio channel, a channel-based loudspeaker format, and a spherical surround sound format.
7. The method of claim 1 , wherein the visual representation is an avatar of a person within a recorded scene represented by the video signal, wherein the person is a sound source for the sound.
8. A non-transitory machine-readable medium storing instructions that, when executed by one or more processors of an electronic device, cause the electronic device to:
receive a visual feature associated with a video signal and an audio feature associated with a sound captured in a microphone signal;
receive metadata comprising a target scene that includes a visual representation of the sound; and
determine, as output of a machine learning (ML) model using 1) the visual feature, 2) the audio feature, and 3) the target scene as input, one or more mapping parameters that, when applied to the microphone signal, generates one or more output audio channels that includes the sound of the microphone signal that is spatially mapped according to a location of the visual representation within the target scene and the mapping parameters are determined based on one or more correlations between the and the visual feature.
9. The non-transitory machine-readable medium of claim 8 , wherein the one or more mapping parameters includes a direction-of-arrival (DoA) estimation of the sound.
10. The non-transitory machine-readable medium of claim 8 , wherein the one or more mapping parameters includes the location of the visual representation of the sound within the target scene.
11. The non-transitory machine-readable medium of claim 8 , wherein the target scene is visually the same as a record scene represented by the video signal.
12. The non-transitory machine-readable medium of claim 8 , wherein the visual representation is a virtual object associated with a recorded object that is a sound source of the sound within a recorded scene of the video signal, wherein the location of the virtual object within the target scene is different than a location of the recorded object within the recorded scene.
13. The non-transitory machine-readable medium of claim 8 , wherein the one or more output audio channels are associated with a target output audio format that is one of: a binaural audio format comprising a left audio channel and a right audio channel, a channel-based loudspeaker format, and a spherical surround sound format.
14. An electronic device comprising:
at least one processor; and
memory having instructions stored therein which when executed by the at least one processor causes the electronic device to:
receive input audio data that includes a sound, video data, and metadata comprising a target scene that includes a visual representation of the sound; and
generate output audio data in a target output audio format as output of a machine learning (ML) model using 1) the input audio data, 2) the video data, and 3) the target scene as input, wherein the ML model maps the input audio data to the output audio data according to the target output audio format, wherein the output audio data comprises the sound that is spatially mapped according to a location of the visual representation within the target scene, wherein the ML model outputs the output audio data based on one or more correlations between the sound and visual information of the video data.
15. The electronic device of claim 14 , wherein the target output audio format is defined in the metadata.
16. The electronic device of claim 14 , wherein the target output audio format comprises at least one of: a binaural audio format, a channel-based loudspeaker format, and a spherical surround sound format.
17. The electronic device of claim 14 , wherein the target output audio format is a first audio format, wherein the input audio data is in a second audio format that is different than the first audio format.
18. The electronic device of claim 14 , wherein the target scene is visually the same as a scene represented by the video data.
19. The electronic device of claim 14 , wherein the visual representation is a virtual object associated with a recorded object that is a sound source of the sound within a recorded scene represented by the video data, wherein the location of the virtual object within the target scene is different than a location of the recorded object within the recorded scene.
20. The electronic device of claim 14 , wherein the output audio data is generated based on mapping the sound of the input audio data to the location of the visual representation within the target scene that is defined in the metadata.