Method of audio-assisted field of view prediction for spherical video streaming
Aspects of the subject disclosure may include, for example, a method for spherical video streaming over a network by a processing system including a processor, the method including prefetching audio data for rendering the spherical video; determining whether there is a single source or a plurality of sources of audio in the audio data; responsive to a determination that there is only one dominant source of the audio in the audio data, prefetching video data in a field of view centered on spatial coordinates of the dominant source of audio; responsive to a determination that there is the plurality of sources of the audio in the audio data, sorting the plurality of sources of the audio according to an estimated sound source energy for each source in the plurality of sources, thereby creating a sorted list of the plurality of sources; and prefetching video data in plural fields of view centered on spatial coordinates of each audio source in the sorted list of the plurality of sources, wherein the prefetching is performed in an order corresponding to the sorted list, and wherein the prefetching occurs within an available bandwidth of the network. Other embodiments are disclosed.
1. A method comprising:
downloading, by a processing system including a processor, audio data for rendering a spherical video;
sorting, by the processing system, a plurality of sources of audio in the audio data according to an estimated sound source energy for each source in the plurality of sources, thereby creating a first sorted list of the plurality of sources;
determining, by the processing system, that a first source and a second source in the first sorted list of the plurality of sources have approximately a same amount of energy and different spatial locations;
determining, by the processing system and based on the determining that the first source and the second source in the first sorted list of the plurality of sources have approximately the same amount of energy and different spatial locations, that a third, dominant source in the first sorted list of the plurality of sources has a location that is nearer to the first source than the second source;
ordering, by the processing system and based on the determining that the third, dominant source in the first sorted list of the plurality of sources has a location that is nearer to the first source than the second source, the first source higher than the second source in a second sorted list of the plurality of sources, wherein the second sorted list of the plurality of sources is based on the first sorted list of the plurality of sources; and
prefetching, by the processing system and in an order of the second sorted list of the plurality of sources, video data in plural fields of view centered on spatial coordinates of each audio source in the second sorted list of the plurality of sources, wherein the prefetching retrieves as many of the plural fields of view within an available bandwidth of a network.
2. The method of claim 1 , wherein the spherical video is streamed over the network.
3. The method of claim 1 , further comprising:
rendering, by the processing system, the audio data and the video data corresponding to a selected field of view, wherein the selected field of view is in the plural fields of view.
4. The method of claim 1 , wherein the audio data comprises an object-based representation that includes metadata that describes the spatial coordinates of each audio source.
5. The method of claim 4 , wherein the audio data conforms to an International Telecommunication Union Radiocommunication sector Audio Definition Model.
6. The method of claim 1 , further comprising:
analyzing, by the processing system, an audio sound field in the audio data to derive spatial coordinates of each sound source.
7. The method of claim 1 , further comprising:
sorting audio sources having approximately a same amount of energy based on their proximity to a viewer, and limiting, by the processing system, the second sorted list to 24 or fewer sources of the audio in the plurality of sources.
8. The method of claim 1 , wherein the estimated sound source energy for each source of the audio is frequency-based.
9. The method of claim 8 , wherein the estimated sound source energy for each source of the audio is based on sound pressure.
10. The method of claim 8 , wherein the estimated sound source energy for each source of the audio is based on a particle velocity.
11. The method of claim 1 , wherein the estimated sound source energy for each source of the audio is based on a steered response power.
12. The method of claim 1 , wherein the estimated sound source energy for each source of the audio is based on an intensity vector.
13. A device, comprising:
a processing system including a processor; and
a memory that stores executable instructions that, when executed by the processing system, facilitate performance of operations, the operations comprising:
downloading audio data associated with a spherical video;
sorting a plurality of sources of audio in the audio data according to criteria for each source in the plurality of sources, thereby creating a first sorted list of the plurality of sources;
determining that a first source and a second source in the first sorted list of the plurality of sources have approximately a same amount of energy and different spatial locations;
determining that a third, dominant source in the first sorted list of the plurality of sources has a location that is nearer to the first source than the second source;
ordering, based on the determining that the third, dominant source in the first sorted list of the plurality of sources has a location that is nearer to the first source than the second source, the first source higher than the second source in a second sorted list of the plurality of sources, wherein the second sorted list of the plurality of sources is based on the first sorted list of the plurality of sources; and
prefetching, in an order of the second sorted list of the plurality of sources, video data in plural fields of view centered on spatial coordinates of each audio source in the second sorted list of the plurality of sources, wherein the prefetching of the video data is terminated after a certain number of fields of view; and
rendering the audio data and the video data corresponding to a selected field of view, wherein the selected field of view is in the plural fields of view.
14. The device of claim 13 , wherein the operations further comprise:
identifying the plurality of sources of audio in accordance with a steered response power.
15. The device of claim 13 , wherein the operations further comprise:
identifying the plurality of sources of audio in accordance with an intensity vector.
16. The device of claim 13 , wherein the criteria comprises relative sound source energy of each source of the audio.
17. A non-transitory, machine-readable medium, comprising executable instructions that, when executed by a processing system including a processor, facilitate performance of operations, the operations comprising:
prefetching audio data from a spherical video;
sorting a plurality of sources of audio in the audio data according to criteria for each source in the plurality of sources, thereby creating a first sorted list of the plurality of sources;
determining that a first source and a second source in the first sorted list of the plurality of sources have approximately a same amount of energy and different spatial locations;
determining that a third, dominant source in the first sorted list of the plurality of sources has a location that is nearer to the first source than the second source;
ordering, based on the determining that the third, dominant source in the first sorted list of the plurality of sources has a location that is nearer to the first source than the second source, the first source higher than the second source in a second sorted list of the plurality of sources, wherein the second sorted list of the plurality of sources is based on the first sorted list of the plurality of sources; and
prefetching video data in an order of the second sorted list of the plurality of sources in plural fields of view centered on spatial coordinates of each audio source in the second sorted list of the plurality of sources; and
rendering the audio data and the video data corresponding to a selected field of view, wherein the selected field of view is in the plural fields of view.
18. The non-transitory, machine-readable medium of claim 17 , wherein the operations further comprise:
analyzing an audio sound field in the audio data to derive spatial coordinates of each source of the audio.
19. The non-transitory, machine-readable medium of claim 18 , wherein the analyzing further comprises a frequency-based computation of a time difference of arrival, a sound pressure, a particle velocity, or a combination thereof.
20. The non-transitory, machine-readable medium of claim 17 , wherein the audio data is associated with an audio stream, wherein the rendering of the audio data corresponds to a binaural rendering that is based on a head-related transfer function (HRTF), and wherein the HRTF incorporates audio filters that are applied to the audio stream so that a listener perceives sounds that come from a particular location of a three-dimensional (3D) space.