Method of audio-assisted field of view prediction for spherical video streaming
Aspects of the subject disclosure may include, for example, a method for spherical video streaming over a network by a processing system including a processor, the method including prefetching audio data for rendering the spherical video; determining whether there is a single source or a plurality of sources of audio in the audio data; responsive to a determination that there is only one dominant source of the audio in the audio data, prefetching video data in a field of view centered on spatial coordinates of the dominant source of audio; responsive to a determination that there is the plurality of sources of the audio in the audio data, sorting the plurality of sources of the audio according to an estimated sound source energy for each source in the plurality of sources, thereby creating a sorted list of the plurality of sources; and prefetching video data in plural fields of view centered on spatial coordinates of each audio source in the sorted list of the plurality of sources, wherein the prefetching is performed in an order corresponding to the sorted list, and wherein the prefetching occurs within an available bandwidth of the network. Other embodiments are disclosed.
1. A method comprising:
determining, by a processing system including a processor, that a first source and a second source in a first sorted list of a plurality of sources of audio have approximately a same amount of energy and different spatial locations;
determining, by the processing system and based on the determining that the first source and the second source in the first sorted list of the plurality of sources of audio have approximately the same amount of energy and different spatial locations, that a third, dominant source in the first sorted list of the plurality of sources of audio has a location that is nearer to the first source than the second source;
ordering, by the processing system and based on the determining that the third, dominant source in the first sorted list of the plurality of sources of audio has a location that is nearer to the first source than the second source, the first source higher than the second source in a second sorted list of the plurality of sources of audio, wherein the second sorted list of the plurality of sources of audio is based on the first sorted list of the plurality of sources of audio; and
prefetching, by the processing system and in an order of the second sorted list of the plurality of sources audio, video data.
2. The method of claim 1 , wherein the video data pertains to a spherical video that is streamed over a network.
3. The method of claim 1 , further comprising:
rendering, by the processing system, audio data and the video data corresponding to a selected field of view.
4. The method of claim 3 , wherein the prefetching comprises prefetching the video data in plural fields of view centered on spatial coordinates for each audio source in the second sorted list of the plurality of sources of audio.
5. The method of claim 4 , wherein the audio data comprises an object-based representation that includes metadata that describes the spatial coordinates of each audio source in the second sorted list of the plurality of sources of audio.
6. The method of claim 5 , wherein the audio data conforms to an International Telecommunication Union Radiocommunication sector Audio Definition Model.
7. The method of claim 4 , further comprising:
analyzing, by the processing system, an audio sound field in the audio data to derive the spatial coordinates.
8. The method of claim 1 , further comprising:
sorting, by the processing system, audio sources having approximately a same amount of energy based on their proximity to a viewer; and
limiting, by the processing system, the second sorted list of the plurality of sources of audio to 24 or fewer sources of audio.
9. The method of claim 1 , wherein the first sorted list of the plurality of sources of audio is based on an estimated sound source energy for each source.
10. The method of claim 9 , wherein the estimated sound source energy for each source is frequency-based.
11. The method of claim 9 , wherein the estimated sound source energy for each source is based on sound pressure.
12. The method of claim 9 , wherein the estimated sound source energy for each source is based on a particle velocity.
13. The method of claim 9 , wherein the estimated sound source energy for each source is based on a steered response power.
14. The method of claim 9 , wherein the estimated sound source energy for each source is based on an intensity vector.
15. A device, comprising:
a processing system including a processor; and
a memory that stores executable instructions that, when executed by the processing system, facilitate performance of operations, the operations comprising:
determining that a first source and a second source in a first sorted list of a plurality of sources of audio have approximately a same amount of energy and different spatial locations;
determining that a third, dominant source in the first sorted list of the plurality of sources of audio has a location that is nearer to the first source than the second source;
ordering, based on the determining that the third, dominant source in the first sorted list of the plurality of sources of audio has a location that is nearer to the first source than the second source, the first source higher than the second source in a second sorted list of the plurality of sources of audio, wherein the second sorted list of the plurality of sources of audio is based on the first sorted list of the plurality of sources of audio;
prefetching, in an order of the second sorted list of the plurality of sources of audio, video data; and
rendering audio data and the video data corresponding to a selected field of view.
16. The device of claim 15 , wherein the operations further comprise:
identifying the plurality of sources of audio in accordance with a steered response power.
17. The device of claim 15 , wherein the operations further comprise:
identifying the plurality of sources of audio in accordance with an intensity vector.
18. A non-transitory, machine-readable medium, comprising executable instructions that, when executed by a processing system including a processor, facilitate performance of operations, the operations comprising:
determining that a first source and a second source in a first sorted list of a plurality of sources of audio have approximately a same amount of energy and different spatial locations;
determining that a third, dominant source in the first sorted list of the plurality of sources of audio has a location that is nearer to the first source than the second source;
ordering, based on the determining that the third, dominant source in the first sorted list of the plurality of sources of audio has a location that is nearer to the first source than the second source, the first source higher than the second source in a second sorted list of the plurality of sources of audio; and
prefetching video data in an order of the second sorted list of the plurality of sources of audio in plural fields of view centered on spatial coordinates of each audio source in the second sorted list of the plurality of sources of audio.
19. The non-transitory, machine-readable medium of claim 18 , wherein the operations further comprise:
analyzing an audio sound field to derive spatial coordinates of each source of audio in the plurality of sources of audio.
20. The non-transitory, machine-readable medium of claim 18 , wherein the plurality of sources of audio are included in audio data, wherein the audio data is associated with an audio stream, and wherein the operations further comprise:
rendering the audio data via a binaural rendering that is based on a head-related transfer function (HRTF), wherein the HRTF incorporates audio filters that are applied to the audio stream so that a listener perceives sounds that come from a particular location of a three-dimensional (3D) space.