Encoding content for virtual, augmented, and mixed reality (xR) applications in connectivity-constrained environments
Embodiments of systems and methods for encoding content for virtual, augmented, or mixed reality (xR) applications in connectivity-constrained environments are described. In some embodiments, an Information Handling System (IHS) may be configured to: receive sensor information from a Head-Mounted Device (HMD) worn by a user during execution of an xR application; calculate, based on the sensor information, a priority of each of a plurality of Regions-of-Interest (ROIs) within one or more images produced by a rendering engine; and indicate the priorities to a video encoder, where the video encoder is configured to use each priority to control at least one of: (a) visual fidelity, or (b) resilience of a corresponding ROI transmitted by the IHS to the HMD in a video signal during execution of the xR application.
1. An Information Handling System (IHS), comprising:
a processor; and
a memory coupled to the processor, the memory having program instructions stored thereon that, upon execution by the processor, cause the IHS to:
receive sensor information from a Head-Mounted Device (HMD) worn by a user during execution of a virtual, augmented, or mixed reality (xR) application;
calculate, based on the sensor information, a priority of each of a plurality of Regions-of-Interest (ROIs) within one or more images produced by a rendering engine, each priority based at least in part, on at least one marker placed in a known stationary location; and
indicate the priorities to a video encoder, wherein the video encoder is configured to use each priority to control at least one of: (a) a visual fidelity level, or (b) a resilience level of a corresponding ROI transmitted by the IHS to the HMD in a video signal during execution of the xR application.
2. The IHS of claim 1 , wherein the sensor information comprises a plurality of Simultaneous Localization And Mapping (SLAM) landmarks.
3. The IHS of claim 2 , wherein a sensor that generates the sensor information comprises an Infra-Red (IR) or Near-IR (NIR) camera mounted on the HMD.
4. The IHS of claim 3 , wherein the SLAM landmark data comprises one or more selected SLAM landmarks, among the plurality of SLAM landmarks, with a Kalman Gain above a threshold value.
5. The IHS of claim 4 , wherein to calculate the priority of each of the plurality of ROIs, the program instructions, upon execution by the processor, further cause the IHS to select a subset of one or more ROIs closest to the SLAM landmarks.
6. The IHS of claim 4 , wherein to indicate the priorities, the program instructions, upon execution by the processor, further cause the IHS to:
map each ROI into a left ROI and a right ROI;
quantize the left and right ROIs; and
provide at least a portion of the quantized left and right ROIs to the video encoder via an Application Programming Interface (API) command.
7. The IHS of claim 6 , wherein the API command is executed by the video encoder to allocate more bandwidth to the transmission of the quantized left and right ROIs relative to other portions of the one or more images in the video signal.
8. The IHS of claim 1 , wherein the sensor information comprises an Eye-Gaze Tracking (EGT) data that is obtained by a gaze vector captured by an EGT camera mounted on the HMD.
9. The IHS of claim 8 , wherein to calculate the priority for each of the plurality of ROIs, the program instructions, upon execution by the processor, further cause the IHS to select a subset of one or more ROIs nearest an object of gaze interaction in the xR application.
10. The IHS of claim 1 , wherein each priority comprises an adaptive intra-refresh rate selected based upon at least one of: (i) a size of a corresponding ROI; or (ii) a measure of packet loss.
11. The IHS of claim 1 , wherein the program instructions, upon execution by the processor, further cause the IHS to:
calculate, based on the sensor information, a priority of each of a plurality of audio ROIs; and
indicate the priorities to an audio encoder, wherein the audio encoder is configured to use each priority to control at least one of: (a) an audio fidelity level, or (b) a resilience level of a corresponding audio ROI transmitted by the IHS to the HMD in an audio signal during execution of the xR application.
12. A method, comprising:
receiving sensor information from a Head-Mounted Device (HMD) worn by a user during execution of a virtual, augmented, or mixed reality (xR) application;
calculating, based on the sensor information, a priority of each of a plurality of audio Regions-of-Interest (ROIs), each priority based at least in part, on at least one marker placed in a known stationary location; and
indicating the priorities to an audio encoder, wherein the audio encoder is configured to use each priority to control at least one of: (a) an audio fidelity level, or (b) a resilience level of a corresponding audio ROI transmitted by the IHS to the HMD in an audio signal during execution of the xR application.
13. The method of claim 12 , wherein the sensor information comprises Simultaneous Localization And Mapping (SLAM) landmarks captured by an Infra-Red (IR) or Near-IR (NIR) camera mounted on the HMD.
14. The method of claim 13 , wherein calculating the priority further comprises selecting one or more SLAM landmarks, among the SLAM landmark data, with a Kalman Gain above a threshold value.
15. The method of claim 13 , further comprising mapping the selected SLAM landmarks to a coordinate space used to reproduce 3D audio and selecting a subset of one or more audio ROIs closest to the SLAM landmarks.
16. The method of claim 12 , wherein the sensor information comprises EGT data that is captured by an Eye-Gaze Tracking (EGT) camera mounted on the HMD.
17. The method of claim 16 , further comprising selecting a subset of one or more ROIs nearest an object of gaze interaction in the xR application.
18. The method of claim 12 , further comprising:
calculating, based on the sensor information, a priority of each of a plurality of ROIs within one or more images produced by a rendering engine; and
indicating the priorities to a video encoder, wherein the video encoder is configured to use each priority to control at least one of: (a) a visual fidelity level, or (b) a resilience level of a corresponding ROI transmitted by the IHS to the HMD in a video signal during execution of the xR application.
19. A non-transitory hardware memory of a Head-Mounted Device (HMD) wearable by a user, the non-transitory hardware memory having program instructions stored thereon that, upon execution by a processor, cause the HMD to:
transmit, to an Information Handling System (IHS), frames captured by a Near-Infrared (NIR) camera mounted on the HMD, wherein the frames are used by the IHS to perform Simultaneous Localization and Mapping (SLAM) of the HMD during execution of a virtual, augmented, or mixed reality (xR) application; and
receive a video signal encoded by the IHS, at least in part, using at least one marker detected in the frames, the video signal comprising a plurality of Regions-of-Interest (ROIs) each including a priority calculated by the IHS, each priority based at least in part, on the at least one marker, wherein the marker is placed in a known stationary location.