SELECTIVE AUDIO REPRODUCTION
A system comprising a video display device, an audio output device and a computing device. The computing device may comprise one or more processors configured to (i) determine orientation angles of a spherical video based on an input, (ii) extract a viewport from the spherical video based on the orientation angles, (iii) output the viewport to the video display device, (iv) render a sound field based on the orientation angles and the audio output device and (v) output the sound field to the audio output device. Sound sources that comprise the sound field are adjusted to align with the viewport. The sound sources outside of the viewport are attenuated.
1 . A method for rendering selective audio playback for a spherical field of view comprising the steps of:
(A) receiving a spherical video stream and an immersive sound field stream;
(B) determining a viewport of a user viewing said spherical video stream;
(C) determining a location for one or more audio sources for said immersive sound field stream; and
(D) adjusting an attenuation of each of said audio sources having said location outside of said viewport.
2 . The method according to claim 1 , further comprising the step of:
determining a playback capability of an audio output device, wherein steps (C)-(D) are not performed if said playback capability of said audio output device supports immersive audio rendering.
3 . The method according to claim 2 , wherein said immersive audio rendering comprises implementing at least one of (a) binaural processing, (b) multi-speaker surround audio and (c) immersive speaker setups.
4 . The method according to claim 2 , wherein said audio output device is implemented as headphones.
5 . The method according to claim 1 , wherein said audio sources are (a) ambisonic audio in a first mode and (b) object-based audio in a second mode.
6 . The method according to claim 5 , wherein said attenuation of said ambisonic audio is performed by selective decoding and processing to align output audio to said viewport.
7 . The method according to claim 5 , wherein said attenuation of said object-based audio is performed by restricting rendering to objects having coordinates that are within at least one of (a) said viewport and (b) a pre-determined area.
8 . The method according to claim 1 , wherein said attenuation of said audio sources is configured to silence said audio sources that are outside of said viewport.
9 . The method according to claim 1 , wherein said attenuation of said audio sources is configured to reduce a level of said audio sources that are outside of said viewport.
10 . The method according to claim 1 , further comprising the step of:
playing back said audio sources that are within said viewport.
11 . The method according to claim 1 , wherein a sound level of said audio sources having said location within said viewport is adjusted to have equal power to a full sound field of said immersive sound field stream to compensate for level differences.
12 . The method according to claim 1 , wherein determining said location of said audio sources for said immersive sound field stream comprises decoding said sound field stream to at least one of (i) an icosahedron of virtual speakers and (ii) a cube of virtual speakers.
13 . The method according to claim 1 , wherein said viewport of said user is determined based on orientation angles.
14 . The method according to claim 13 , further comprising the steps of:
extracting said viewport from an equirectangular representation of said spherical video stream;
reprojecting said viewport into a rectilinear view adapted to a video display device; and
steering said audio sources using said orientation angles so that an alignment of said audio sources matches said viewport.
15 . The method according to claim 1 , wherein adjusting said attenuation of said audio sources having said location outside of said viewport implements a selective audio playback.
16 . The method according to claim 1 , wherein said audio sources are transformed by rotating said immersive sound field stream.
17 . The method according to claim 1 , wherein (i) adjusting said attenuation of said audio sources that are outside said viewport further comprises implementing audio effects to simulate audio originating behind a viewer and (ii) said audio effects comprise at least one of a reverb and a delay.
18 . A system comprising:
a video display device;
an audio output device; and
a computing device comprising one or more processors configured to (i) determine orientation angles of a spherical video based on an input, (ii) extract a viewport from said spherical video based on said orientation angles, (iii) output said viewport to said video display device, (iv) render a sound field based on said orientation angles and said audio output device and (v) output said sound field to said audio output device, wherein (a) sound sources that comprise said sound field are adjusted to align with said viewport and (b) said sound sources outside of said viewport are attenuated.
19 . A system comprising:
a video source configured to generate a spherical video stream;
a plurality of audio sources each configured to generate audio signals; and
a computing device comprising one or more processors configured to (i) determine orientation angles of said spherical video stream based on an input, (ii) extract a viewport from said spherical video stream based on said orientation angles, (iii) output said viewport to a video output stream, (iv) render a sound field based on said orientation angles and an audio output device and (v) output said sound field to an audio output stream, wherein said audio signals that comprise said sound field that are outside of said viewport are attenuated.
20 . The system according to claim 19 , wherein (a) said video output stream is presented to a video encoder and (b) said audio output stream is presented to an audio encoder.