Adaptive backchannel synchronization for virtual, augmented, or mixed reality (xR) applications in edge cloud architectures
Embodiments of systems and methods for adaptive backchannel synchronization for virtual, augmented, or mixed reality (xR) applications in edge cloud architectures are described. In some embodiments, an Information Handling System (IHS) may include a memory having program instructions stored thereon that, upon execution by a processor, cause the IHS to: receive, from a first Head-Mounted Device (HMD), first Simultaneous Localization and Mapping (SLAM) data comprising a first plurality of packets; receive, from the HMD, an indication of a synchronized time; and at least one of: in response to a determination that the first SLAM data is advanced with respect to the synchronized time by a threshold value, drop one or more of the first plurality of packets; or in response to a determination that the first SLAM data is lagged with respect to the synchronized time by the threshold value, transform lagged SLAM data into current SLAM data.
1. An Information Handling System (IHS), comprising:
a processor; and
a memory coupled to the processor, the memory having program instructions stored thereon that, upon execution, cause the IHS to:
receive, from a first Head-Mounted Device (HMD), first Simultaneous Localization and Mapping (SLAM) data comprising a plurality of first packets;
receive, from the first HMD, an indication of a synchronized time;
when one or more of the first packets are advanced with respect to the synchronized time by at least a threshold value, drop the one or more first packets, the threshold value indicating an amount of delay in which the first packets of the SLAM data are determined to be out of synchronization with respect to a second data stream; and
when the one or more first packets are lagged with respect to the synchronized time by at least the threshold value, transform the lagged one or more first packets into current SLAM data.
2. The IHS of claim 1 , wherein the indication of the synchronized time comprises a SLAM frame rate and a seed value.
3. The IHS of claim 1 , wherein each of the plurality of packets includes at least one SLAM landmark and a timestamp.
4. The IHS of claim 3 , wherein each of the first plurality of packets further comprises pose information.
5. The IHS of claim 3 , wherein to transform the lagged first packets into the current SLAM data, the program instructions, upon execution, further cause the IHS to apply a camera transform matrix to the lagged first packets using pose information obtained from an earlier SLAM packet.
6. The IHS of claim 1 , wherein the program instructions, upon execution, further cause the IHS to select the threshold value as: a first value in response to the IHS being a co-located IHS, a second value greater than the first value in response to the IHS being an edge server, or a third value greater than the second value in response to the IHS being a cloud server.
7. The IHS of claim 1 , wherein the threshold value is selected based upon a latency parameter associated with the IHS.
8. The IHS of claim 1 , wherein the threshold value is selected based upon a speed of movement of the first HMD.
9. The IHS of claim 1 , wherein the program instructions, upon execution, further cause the IHS to:
receive, from the first HMD, World-Facing Camera (WFC) or Gesture Recognition and Tracking (GRT) data comprising a plurality of second packets; and
when one or more of the second packets are advanced with respect to the synchronized time by at least the threshold value, drop one or more of the plurality second packets; and
when the one or more second packets are lagged with respect to the synchronized time by at least the threshold value, transform the one or more lagged second packets into current WFC or GRT data using temporal interpolation.
10. The IHS of claim 1 , wherein the program instructions, upon execution, further cause the IHS to:
receive, from the first HMD, Eye-Gaze Tracking (EGT) data comprising a second plurality of packets; and
when one or more of the second packets are advanced with respect to the synchronized time by at least the threshold value, drop the one or more second packets; and
when the one or more second packets are lagged with respect to the synchronized time by at least the threshold value, transform the one or more second packets into current EGT data using linear interpolation.
11. The IHS of claim 1 , wherein the program instructions, upon execution, further cause the IHS to:
receive, from the first HMD, audio data comprising a plurality of second packets; and
when one or more of the second packets are advanced with respect to the synchronized time by at least the threshold value, transform the one or more lagged second packets into current audio data using time scale modulation downsampling; and
when the one or more second packets are lagged with respect to the synchronized time by at least the threshold value, transform the one or more lagged second packets into current audio data using time scale modulation upsampling.
12. The IHS of claim 1 , wherein the program instructions, upon execution, further cause the IHS to:
receive, from a second HMD co-located with the first HMD, second SLAM data comprising a second plurality of packets; and
when one or more of the second packets are advanced with respect to the synchronized time by at least a second threshold value, wherein the second threshold value is larger than the first threshold value, drop one or more of the second packets; and
when the one or more second packets are lagged with respect to the other synchronized time by at least the second threshold value, transform the one or more second packets into current SLAM data.
13. A hardware memory having program instructions stored thereon that, upon execution by an Information Handling System (IHS), cause the IHS to:
receive Simultaneous Localization and Mapping (SLAM) data comprising a plurality of first packets;
select a first threshold value that indicates an amount of delay in which the first packets of the SLAM data are determined to be out of synchronization with respect to a second data stream; and
when one or more of the first packets are advanced by the selected threshold value or more, drop the one or more first packets; and
when the one or more first packets are lagged by the selected threshold value or more, transform the one or more lagged first packets into current SLAM data.
14. The hardware memory of claim 13 , wherein the program instructions, upon execution, further cause the IHS to:
receive World-Facing Camera (WFC) or Gesture Recognition and Tracking (GRT) data comprising a plurality of second packets; and
when one or more of the first packets are advanced with respect to the synchronized time by the threshold value or more, drop the one or more second packets; and
when the one or more second packets are lagged with respect to the synchronized time by the threshold value or more, transform the one or more lagged second packets into current WFC or GRT data.
15. The hardware memory of claim 13 , wherein the program instructions, upon execution, further cause the IHS to:
receive Eye-Gaze Tracking (EGT) data comprising a plurality of second packets; and
when one or more of the second packets are advanced with respect to the synchronized time by the threshold value or more, drop the one or more second packets; or
when the one or more second packets are lagged with respect to the synchronized time by the threshold value or more, transform the one or more lagged second packets into current EGT data.
16. The hardware memory of claim 13 , wherein the program instructions, upon execution, further cause the IHS to:
receive audio data; and
when the audio data is advanced or lagged with respect to the synchronized time by the threshold value or more, transform the lagged audio data into current audio data.
17. A method, comprising:
receiving, by an Information Handling System (IHS) from a Head-Mounted Device (HMD), Simultaneous Localization and Mapping (SLAM) data comprising a plurality of first packets; and
when one or more of the first packets are advanced by at least a first threshold value, dropping the one or more first packets, the first threshold value indicating an amount of delay in which the first packets of the SLAM data are determined to be out of synchronization with respect to a second data stream; and
when the one or more first packets are lagged by at least the first threshold value, transforming the one or more lagged first packets into current SLAM data.
18. The method of claim 17 , further comprising:
receiving, from the HMD, World-Facing Camera (WFC) or Gesture Recognition and Tracking (GRT) data comprising a plurality of second packets; and
when one or more of the second packets are advanced by at a second threshold value, dropping the one or more second packets; or
when the one or more second packets are lagged by at least the second threshold value, transforming the one or more lagged second packets into current WFC or GRT data.
19. The method of claim 18 , further comprising:
receiving, from the HMD, Eye-Gaze Tracking (EGT) data comprising a plurality of third packets; and
when one or more of the third packets are advanced by at least a third threshold value, dropping the one or more of third packets; and
when the one or more third packets are lagged with respect to the synchronized time by at least the third threshold value, transforming the one or more lagged third packets into current EGT data.
20. The method of claim 19 , further comprising:
receiving, from the HMD audio data comprising a plurality of fourth packets; and
when one or more of the fourth packets are advanced or lagged with by at least a fourth threshold value, synchronizing the one or more fourth packets with the SLAM data.