Streaming scene prioritizer for immersive media
Aspects of the disclosure provide methods and apparatuses for immersive media processing. In some examples, a method of media processing includes receiving a scene based immersive media for playing at a light field based display. The scene based immersive media includes a plurality of scenes. The method includes assigning priority values respectively to the plurality of scenes in the scene based immersive media, and determining an order for streaming the plurality of scenes to an end device according to the priority values. In some examples, a method of media processing includes receiving a scene based immersive media for playing at a light field based display. A scene in the scene based immersive media includes a plurality of assets in a first order. The method includes determining a second order for streaming the plurality of assets to an end device, the second order is different from the first order.
1 . A method of immersive media processing, comprising:
receiving, by a network device, a scene based immersive media for display by a light field based display, the scene based immersive media comprising a plurality of immersive media scenes;
assigning priority values and complexity values respectively to the plurality of immersive media scenes based on one or more attributes of the plurality of immersive media scenes such that the plurality of immersive media scenes is ordered in a first order based on the priority values and the complexity values;
transmitting the plurality of immersive media scenes in the first order to an end device;
determining a second order for streaming the plurality of immersive media scenes to the end device by adjusting the first order of the plurality of immersive media scenes according to updated priority values and updated complexity values that are obtained by updating the priority values and the complexity values respectively based on a feedback signal received from the end device, the second order being different from the first order; and
transmitting the plurality of immersive media scenes to the end device in the second order.
2 . The method of claim 1 , wherein the assigning the priority values further comprises:
assigning the priority values to the plurality of immersive media scenes based on a viewing position of the scene based immersive media of a user that is indicated by the feedback signal.
3 . The method of claim 1 , wherein the determining the second order further comprises:
selecting, by a priority-aware network device, from a subset of un-transmitted scenes in the plurality of immersive media scenes, a highest priority scene with a highest priority value; and
transmitting the highest priority scene to the end device.
4 . The method of claim 1 , wherein the assigning the priority values further comprises:
determining a priority value to a scene based on a likelihood of a need to render the scene.
5 . The method of claim 1 , further comprising:
determining that an available network bandwidth is limited;
selecting, from a subset of un-transmitted scenes in the plurality of immersive media scenes, a highest priority scene with a highest priority value; and
transmitting the highest priority scene when the available network bandwidth is limited.
6 . The method of claim 1 , further comprising:
determining that an available network bandwidth is limited; and
identifying a subset of the plurality of immersive media scenes are unlikely to be needed for rendering next based on the priority values,
wherein the subset of the plurality of immersive media scenes is not streamed when the available network bandwidth is limited.
7 . The method of claim 1 , wherein the assigning the priority values further comprises:
assigning, a first priority value to a first scene based on a second priority value of a second scene according to a relationship between the first scene and the second scene.
8 . The method of claim 1 , further comprising:
receiving, the feedback signal from the end device; and
adjusting at least a priority value of a scene in the plurality of immersive media scenes based on the feedback signal.
9 . The method of claim 8 , further comprising:
assigning, a highest priority to a first scene when the feedback signal indicates a current scene is a second scene that is in connection with the first scene.
10 . The method of claim 8 , wherein the feedback signal indicates a priority adjustment that is determined by the end device.
11 . A method of immersive media processing, comprising:
receiving, by an end device, a media presentation description (MPD) for a scene based immersive media for display by a light field based display, the scene based immersive media comprising a plurality of immersive media scenes, and the MPD being indicative of streaming the plurality of immersive media scenes to the end device in a first order that is determined according to priority values and complexity values of the plurality of immersive media scenes;
sending, by the end device, a feedback signal to a server, updated priority values and updated complexity values of the plurality of immersive media scenes that are determined by updating the priority values and the complexity values respectively based on the feedback signal;
receiving, by the end device, the plurality of immersive media scenes in a second order that is determined according to the updated priority values and the updated complexity values, the second order being determined by adjusting the first order of the plurality of immersive media scenes; and
outputting for display the plurality of immersive media scenes of immersive media by the end device based on the second order, the second order being different from the first order.
12 . The method of claim 11 , wherein the feedback signal indicates a next scene to render.
13 . The method of claim 11 , wherein the feedback signal indicates an adjustment of a priority value of at least one scene of the plurality of immersive media scenes.
14 . The method of claim 11 , wherein the feedback signal indicates a viewing position of a user of a current scene.
15 . A method of immersive media processing, comprising:
receiving, by a network device, a scene based immersive media for display by a light field based display, a scene in the scene based immersive media comprising a plurality of assets in a first order that is determined according to priority values and complexity values of the plurality of assets;
transmitting the plurality of assets to an end device based on the first order;
assigning updated priority values and updated complexity values respectively to the plurality of assets in the scene, the updated priority values and the updated complexity values being determined by updating the priority values and the complexity values respectively based on one or more attributes of the plurality of assets and a feedback signal of the end device;
determining a second order for streaming the plurality of assets to the end device according to the updated priority values and the updated complexity values, the second order being different from the first order and being determined by adjusting the first order of the plurality of assets; and
transmitting the plurality of assets to the end device based on the second order.
16 . The method of claim 15 , comprising:
determining the second order for streaming the plurality of assets according to a viewing position of the scene based immersive media of a user.
17 . The method of claim 16 , comprising:
assigning a priority value to an asset in the scene according to a size of the asset.
18 . The method of claim 16 , comprising:
assigning a priority value to an asset in the scene according to a visibility of the asset associated with a default entry location of the scene.
19 . The method of claim 16 , comprising:
assigning a first set of priority values to the plurality of assets based on a first attribute of the plurality of assets, the first set of priority values being used for ordering the plurality of assets in a first prioritization scheme;
assigning a second set of priority values to the plurality of assets based on a second attribute of the plurality of assets, the second set of priority values being used for ordering the plurality of assets in a second prioritization scheme;
selecting, a prioritization scheme for the end device from the first prioritization scheme and the second prioritization scheme according to information of the end device; and
ordering the plurality of assets for streaming according to the selected prioritization scheme.
20 . The method of claim 16 , comprising:
assigning a first priority value to a first asset and a second priority value to a second asset when the first asset has a higher complexity of computation than the second asset, the first priority value being higher than the second priority value.