Forward channel contextual error concealment and sync for virtual, augmented, or mixed reality (XR) content in connectivity-constrained environments
Embodiments of systems and methods for forward channel contextual error concealment and sync for virtual, augmented, or mixed reality (xR) content in connectivity-constrained environments are described. In some embodiments, a Head-Mounted Device (HMD) may include a processor; and a memory coupled to the processor, the memory having program instructions stored thereon that, upon execution by the processor, cause the HMD to: decode content received from an Information Handling System (IHS) coupled to the HMD; and handle an error in the decoded content using context information received from a sensor mounted on the HMD.
1. A Head-Mounted Device (HMD), comprising:
a processor; and
a memory coupled to the processor, the memory having program instructions stored thereon that, upon execution by the processor, cause the HMD to:
decode content received from an Information Handling System (IHS) coupled to the HMD;
determine that motion between video frames in the decoded content is greater or smaller than a threshold; and
handle motion data error in the decoded content using motion data received from a Simultaneous Localization And Mapping (SLAM) sensor mounted on the HMD, wherein the motion data comprises a motion vector, and wherein handling the motion data error comprises concealing the motion data error or reconstructing a video frame, at least in part, using a decoded picture buffer local to the HMD.
2. The HMD of claim 1 , wherein the motion data is calculated using a plurality of SLAM landmarks selected with a Kalman Gain above a threshold value.
3. The HMD of claim 1 , wherein the motion data is calculated using linear and angular motion measurements of the HMD.
4. The HMD of claim 1 , wherein the program instructions, upon execution by the processor, further cause the HMD to transform a coordinate system of the motion vector to another coordinate system used to produce the content.
5. The HMD of claim 1 , wherein the program instructions, upon execution, further cause the HMD to conceal the motion data error or reconstruct a video frame using temporal or spatial interpolation depending upon a magnitude of the motion vector.
6. The HMD of claim 1 , wherein the program instructions, upon execution, further cause the HMD to:
in response to a determination that (i) the HMD has moved and (ii) the decoded content has not changed, conceal the motion data error or reconstruct another video frame, at least in part, using a Motion Smoothing (MS) buffer local to the HMD.
7. The HMD of claim 1 , wherein the program instructions, upon execution, further cause the HMD to:
in response to a determination that (i) the HMD has moved and (ii) the decoded content has changed, conceal the motion data error or reconstruct another video frame, at least in part, using an Asynchronous Reprojection (ARP) buffer local to the HMD.
8. A method, comprising:
decoding content received from an Information Handling System (IHS) coupled to a Head-Mounted Device (HMD);
determining that motion between video frames in the decoded content is greater or smaller than a threshold; and
handling motion data error in the decoded content using motion data received from a Simultaneously And Mapping (SLAM) sensor mounted on the HMD, wherein the motion data comprises a motion vector, and wherein handling the motion data error comprises concealing the motion data error or reconstructing a video frame, at least in part, using a Motion Smoothing (MS) buffer local to the HMD.
9. The method of claim 8 , wherein the motion data error comprises a synchronization error between: (i) audio data and video data, (ii) audio data and controller data, or (iii) video data and controller data.
10. The method of claim 8 , wherein handling the motion data error further comprises selecting a synchronization tolerance based upon the motion data.
11. The method of claim 8 , further comprising performing a video frame temporal interpolation operation to catch the video data up to controller data.
12. The method of claim 8 , further comprising performing a time scale modulation operation to catch up or lag audio data.
13. The method of claim 8 , wherein handling the motion data error comprises handling the motion data error in response to another determination that: (i) the HMD has moved, and (ii) the decoded content has or has not changed.
14. The method of claim 8 , further comprising handling the motion data error in the decode content using context information extracted from the decoded content.
15. The method of claim 8 , further comprising handling the motion data error in the decode content using context information comprising a current mode of operation of the HMD.
16. A hardware memory of a Head-Mounted Device (HMD) wearable by a user, the hardware memory having program instructions stored thereon that, upon execution by a processor, cause the HMD to:
decode content received from an Information Handling System (IHS) coupled to the HMD;
determine that motion between video frames in the decoded content is greater or smaller than a threshold; and
handle motion data error in the decoded content using motion data received from a Simultaneously And Mapping (SLAM) sensor mounted on the HMD, wherein the motion data comprises a motion vector, and wherein handling the motion data error comprises concealing the motion data error or reconstructing a video frame, at least in part, using an Asynchronous Reprojection (ARP) buffer local to the HMD.
17. The hardware memory of claim 16 , wherein handling the motion data error comprises handling the motion data error in response to another determination that: (i) the HMD has moved, and (ii) the decoded content has or has not changed.
18. The hardware memory of claim 16 , wherein the program instructions, upon execution, further cause the HMD to select a synchronization tolerance based upon the motion data.
19. The hardware memory of claim 16 , wherein the program instructions, upon execution, further cause the HMD to perform a video frame temporal interpolation operation to catch the video data up to controller data.
20. The hardware memory of claim 16 , wherein the program instructions, upon execution, further cause the HMD to perform a time scale modulation operation to catch up or lag audio data.
21. The hardware memory of claim 16 , wherein to handle the motion data error, the program instructions cause the HMD to use context information extracted from the decoded content.
22. The hardware memory of claim 16 , wherein to handle the motion data error, the program instructions cause the HMD to use a current mode of operation of the HMD.