Adaptive resolution of point cloud and viewpoint prediction for video streaming in computing environments
A mechanism is described for facilitating adaptive resolution and viewpoint-prediction for immersive media in computing environments. An apparatus of embodiments, as described herein, includes one or more processors to receive viewing positions associated with a user with respect to a display, and analyze relevance of media contents based on the viewing positions, where the media content includes immersive videos of scenes captured by one or more cameras. The one or more processors are further to predict portions of the media contents as relevant portions based on the viewing positions and transmit the relevant portions to be rendered and displayed.
1. An apparatus comprising:
processor circuitry coupled to a memory, the processor circuitry to:
generate physical displacement information associated with a user based on physical displacements associated with the user;
transmit the physical displacement information to be evaluated for estimating future physical displacements of the user;
receive, based on the future physical displacements, encoded information of media contents to be rendered by a display, wherein the encoded information includes point clouds and/or textured triangles and prediction information relating to objects and/or regions with respect to low fidelity level or high fidelity level; and
prioritize, during rendering of the encoded information for display by the display, execution times for the objects and/or the regions with the high fidelity level.
2. The apparatus of claim 1 , wherein the processor circuitry is further to track the physical displacements associated with the user with respect to real world.
3. The apparatus of claim 1 , wherein the prediction information is determined based on future pose information, and the future pose information is determined based on one or more viewing positions of the user.
4. The apparatus of claim 1 , wherein the objects and/or the regions with high fidelity are predicted to be more likely to be viewed by the user based on one or more of future head pose information and the future physical displacements, and the future head pose information comprises one or more of future head positions or movements of a head of the user with respect to the display, future user-visible objects or regions of the media contents, and future user-interested objects or regions of the media contents.
5. A method, comprising:
receiving physical displacement information associated with a user;
estimating future physical displacements of the user within a virtual immersive environment based on the physical displacement information;
determining that an object has a relevance level based on the future physical displacements of the user and object metadata of the virtual immersive environment;
associating the object with a fidelity label that corresponds to the relevance level; and
encoding a point cloud representing the object according to the fidelity label associated with the object to obtain an encoded point cloud.
6. The method of claim 5 , further comprising:
prioritizing, during rendering of the encoded point cloud for display, execution for the object with the fidelity label.
7. The method of claim 5 , wherein encoding the point cloud representing the object comprises:
identifying a subset of patches of the point cloud representing the object based on the future physical displacements of the user; and
encoding the subset of patches.
8. The method of claim 7 , wherein identifying the subset of patches comprises applying a texel space shading technique.
9. The method of claim 5 , further comprising:
determining a viewport based on the future physical displacements of the user;
wherein encoding the point cloud representing the object further comprises:
packing patches of the point cloud in a two-dimensional rectangle; and
selecting a subset of patches to recreate a rendered texture of the object for the viewport.
10. The method of claim 5 , wherein encoding the point cloud representing the object comprises:
applying adaptive multi-frequency shading rendering pass to the point cloud representing the object based on the future physical displacements of the user to produce a sparse texture of the point cloud; and
encoding the sparse texture.
11. The method of claim 10 , wherein applying the adaptive multi-frequency shading rendering pass comprises:
determining a level of detail for a given position in object space based on the future physical displacements of the user to reduce a bandwidth of the encoded point cloud.
12. The method of claim 10 , wherein encoding the point cloud representing the object further comprises:
compressing the sparse texture.
13. The method of claim 5 , wherein encoding the point cloud representing the object further comprises:
varying a resolution of the point cloud based on the fidelity label associated with the object.
14. The method of claim 5 , wherein determining that the object has the relevance level comprises:
determining that the object is covered or occluded based on the future physical displacements of the user.
15. The method of claim 5 , wherein determining that the object has the relevance level comprises:
determining that the object is a pre-defined object of interest.
16. The method of claim 5 , wherein determining that the object has the relevance level comprises:
determining that the object is a person's face.
17. One or more non-transitory computer-readable media having instructions stored thereon, which, when executed, cause a computing device to perform operations comprising:
receiving physical pose information associated with a user;
estimating a future head pose of the user within a virtual immersive environment based on the physical pose information;
determining that an object has a relevance level based on the future head pose of the user;
associating the object with a fidelity label that corresponds to the relevance level; and
encoding a point cloud representing the object according to the fidelity label associated with the object to generate an encoded point cloud.
18. The one or more non-transitory computer-readable media of claim 17 , wherein the operations further comprise:
prioritizing, during rendering of the encoded point cloud for display, execution for the object with the fidelity label.
19. The one or more non-transitory computer-readable media of claim 17 , wherein encoding the point cloud representing the object comprises:
identifying a subset of patches of the point cloud representing the object based on the future head pose of the user; and
encoding the subset of patches.
20. The one or more non-transitory computer-readable media of claim 17 , wherein encoding the point cloud representing the object comprises:
varying a resolution of the point cloud based on the fidelity label associated with the object.