IP Library › Granted Patent US 12,231,483
Granted Patent B1
US 12,231,483 · App. 18/234,548 · Granted Feb 18, 2025

Extended and mixed reality experience adaptation, processing, and handover from one edge node to another

Inventors: Tatu V. J. Harviainen (Helsinki, FI); Tao Chen (Palo Alto, CA); Dhananjay Lal (Englewood, CO)
Assignee: Adeia Guides Inc.
H04L65/80G06F3/016G06T19/00H04L65/1069G06T2200/16G06T2210/61
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,231,483
App. No.
18/234,548
Granted
Feb 18, 2025
Kind
B1
Abstract

Methods and systems are described for extended and mixed reality experience adaptation, processing, and handover from one computing or processing entity to another. In response to a change in a condition impacting resource usage, delivery of content is transferred from one edge node to another. One or more changes in a viewing client, a first edge node, a second edge node, a communication network, and content signal a handover. Artificial intelligence systems, including neural networks, and models are trained and developed for improving the adaptation, processing, and handover. Related apparatuses, devices, techniques, and articles are also described.

Claims (95)

1. A method, comprising:

modifying, at a first edge node, a content cache and a digital replica of a local environment based on extended reality (XR) content and a global anchoring map;

transmitting, from the first edge node, data of the content cache and the digital replica of the local environment to a viewing client; and

based at least in part on a change in a condition impacting resource usage, transferring at least a portion of the modifying of the content cache and the digital replica of the local environment based on the XR content and the global anchoring map from the first edge node to a second edge node, wherein the second edge node is configured to perform:

modifying, at the second edge node, the content cache and the digital replica of the local environment based on the XR content and the global anchoring map; and

transmitting, from the second edge node, the data of the content cache and the digital replica of the local environment to the viewing client.

2. The method of claim 1 , comprising:

generating, at a content authoring tool, an asset and a scene description;

transmitting the asset and the scene description to a content server;

generating, at the content server, the extended reality (XR) content and the global anchoring map based on the asset and the scene description;

when the transferring the at least the portion of the modifying from the first edge node to the second edge node has not occurred, transmitting, from the content server, the XR content in accordance with the global anchoring map to the first edge node; and

when the transferring the at least the portion of the modifying from the first edge node to the second edge node has occurred, transmitting, from the content server, the XR content in accordance with the global anchoring map to the second edge node.

3. The method of claim 2 , comprising:

performing content pre-processing by:

generating, at the content authoring tool, the XR content, rules of adaptation, and the global anchoring map;

generating, at the content authoring tool, XR scene data and metadata based on the XR content, the rules of adaptation, and the global anchoring map; and

transmitting, from the content authoring tool, the XR scene data and the metadata to the content server.

4. The method of claim 3 , comprising:

performing content streaming by:

receiving a content request from a user device;

transmitting the content request from the user device to the viewing client;

in response to receiving the content request, collecting, at the viewing client, device capabilities and session conditions;

receiving user input from the user device;

transmitting the user input from the user device to the viewing client;

in response to receiving the user input, processing, at the viewing client, the user input, and collecting device sensor data;

transmitting, from the viewing client, the device capabilities and the session conditions to the first edge node when transferring has not occurred or to the second edge node when transferring has occurred;

transmitting, from the viewing client, the user input and the sensor data to the first edge node when transferring has not occurred or to the second edge node when transferring has occurred;

adapting, at the first edge node or the second edge node, the XR content;

transmitting, from the first edge node or the second edge node, the content request to the content server;

selecting, at the content server, the XR content based on a location of the user device and the global anchoring map;

transmitting, from the content server, the XR scene data and the metadata to the first edge node or the second edge node;

updating, at the first edge node or the second edge node, the XR scene data, and rendering a view from a standpoint of the user device;

transmitting, from the first edge node or the second edge node, streaming output to the viewing client; and

transmitting, from the viewing client, the streaming output to the user device.

5. The method of claim 2 , wherein the content server performs:

receiving a content request from the first edge node or the second edge node;

in response to receiving the request and the content request from the first edge node or the second edge node, determining whether the content request is based on a location;

in response to determining that the content request is based on the location, sending the XR content based on the global anchoring map; and

in response to determining that the content request is not based on the location, sending the XR content based on the content request.

6. The method of claim 1 , wherein the viewing client is configured to monitor device capabilities.

7. The method of claim 6 , wherein at least one of a haptics module, a graphics module, a camera module, an eye tracking module, or a user input module is configured to communicate with the viewing client, and

wherein the viewing client is configured to monitor the device capabilities of the at least one of the haptics module, the graphics module, the camera module, the eye tracking module, or the user input module.

8. The method of claim 1 , wherein the viewing client is configured to transmit output to at least one of a graphics output module or a haptics output module.

9. The method of claim 1 , wherein the change in the condition impacting the resource usage includes at least one of a change in a first condition of the viewing client, a change in a second condition of the first edge node, a change in a third condition of the second edge node, a change in a fourth condition of a communication network linking the viewing client with the first edge node and/or the second edge node, or a change in a fifth condition of the content.

10. The method of claim 1 , wherein at least the first edge node or the second edge node is configured to perform session run-time processing based on a spatial computing server.

11. A system, comprising:

a first edge node configured to modify a content cache and a digital replica of a local environment based on extended reality (XR) content and a global anchoring map;

a viewing client; and

a second edge node configured to modify the content cache and the digital replica of the local environment based on the XR content and the global anchoring map; and

circuitry configured to:

transmit, from the first edge node, data of the content cache and the digital replica of the local environment to the viewing client;

based at least in part on a change in a condition impacting resource usage, transfer at least a portion of the modifying of the content cache and the digital replica of the local environment based on the XR content and the global anchoring map from the first edge node to the second edge node;

modify, at the second edge node, the content cache and the digital replica of the local environment based on the XR content and the global anchoring map; and

transmit, from the second edge node, the data of the content cache and the digital replica of the local environment to the viewing client.

12. The system of claim 11 , comprising:

a content authoring tool configured to generate an asset and a scene description; and

a content server configured to generate the extended reality (XR) content and the global anchoring map based on the asset and the scene description,

wherein the circuitry is configured to:

transmit the asset and the scene description to the content server;

when the transferring the at least the portion of the modifying from the first edge node to the second edge node has not occurred, transmit, from the content server, the XR content in accordance with the global anchoring map to the first edge node; and

when the transferring the at least the portion of the modifying from the first edge node to the second edge node has occurred, transmit, from the content server, the XR content in accordance with the global anchoring map to the second edge node.

13. The system of claim 12 , wherein the circuitry is configured to:

perform content pre-processing by:

generating, at the content authoring tool, the XR content, rules of adaptation, and the global anchoring map;

generating, at the content authoring tool, XR scene data and metadata based on the XR content, the rules of adaptation, and the global anchoring map; and

transmitting, from the content authoring tool, the XR scene data and the metadata to the content server.

14. The system of claim 13 , wherein the circuitry is configured to:

perform content streaming by:

receiving a content request from a user device;

transmitting the content request from the user device to the viewing client;

in response to receiving the content request, collecting, at the viewing client, device capabilities and session conditions;

receiving user input from the user device;

transmitting the user input from the user device to the viewing client;

in response to receiving the user input, processing, at the viewing client, the user input, and collecting device sensor data;

transmitting, from the viewing client, the device capabilities and the session conditions to the first edge node when transferring has not occurred or to the second edge node when transferring has occurred;

transmitting, from the viewing client, the user input and the sensor data to the first edge node when transferring has not occurred or to the second edge node when transferring has occurred;

adapting, at the first edge node or the second edge node, the XR content;

transmitting, from the first edge node or the second edge node, the content request to the content server;

selecting, at the content server, the XR content based on a location of the user device and the global anchoring map;

transmitting, from the content server, the XR scene data and the metadata to the first edge node or the second edge node;

updating, at the first edge node or the second edge node, the XR scene data, and rendering a view from a standpoint of the user device;

transmitting, from the first edge node or the second edge node, streaming output to the viewing client; and

transmitting, from the viewing client, the streaming output to the user device.

15. The system of claim 12 , wherein the content server performs:

receiving a content request from the first edge node or the second edge node;

in response to receiving the request and the content request from the first edge node or the second edge node, determining whether the content request is based on a location;

in response to determining that the content request is based on the location, sending the XR content based on the global anchoring map; and

in response to determining that the content request is not based on the location, sending the XR content based on the content request.

16. The system of claim 11 , wherein the viewing client is configured to monitor device capabilities.

17. The system of claim 16 , comprising:

at least one of a haptics module, a graphics module, a camera module, an eye tracking module, or a user input module configured to communicate with the viewing client,

wherein the viewing client is configured to monitor the device capabilities of the at least one of the haptics module, the graphics module, the camera module, the eye tracking module, or the user input module.

18. The system of claim 11 , wherein the viewing client is configured to transmit output to at least one of a graphics output module or a haptics output module.

19. The system of claim 11 , wherein the change in the condition impacting the resource usage includes at least one of a change in a first condition of the viewing client, a change in a second condition of the first edge node, a change in a third condition of the second edge node, a change in a fourth condition of a communication network linking the viewing client with the first edge node and/or the second edge node, or a change in a fifth condition of the content.

20. The system of claim 11 , wherein at least the first edge node or the second edge node is configured to perform session run-time processing based on a spatial computing server.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 17, 2023
From: HARVIAINEN, TATU V.J.; CHEN, TAO; LAL, DHANANJAY
To: ADEIA GUIDES INC.
Reel/Frame 064623/0770 →
References Cited (55)
US 7990374B2 · Itkowitz et al. · 2011 [cited by applicant]
US 11423624B2 · Dedonato · 2022 [cited by examiner]
US 11670014B2 · Kreiner · 2023 [cited by examiner]
US 11924216B2 · Thomas · 2024 [cited by applicant]
US 20150302665A1 · Miller · 2015 [cited by applicant]
US 20150334430A1 · Clapp · 2015 [cited by applicant]
US 20170075701A1 · Ricci et al. · 2017 [cited by applicant]
US 20180084305A1 · Sprenger et al. · 2018 [cited by applicant]
US 20180227063A1 · Heubel et al. · 2018 [cited by applicant]
US 20190208007A1 · Khalid · 2019 [cited by applicant]
US 20200026354A1 · Swindells · 2020 [cited by applicant]
US 20200142484A1 · Maalouf et al. · 2020 [cited by applicant]
US 20210150818A1 · Dedonato · 2021 [cited by examiner]
US 20230105481A1 · Kreiner · 2023 [cited by examiner]
US 20230249064A1 · Murphy et al. · 2023 [cited by applicant]
US 20230341941A1 · Clark et al. · 2023 [cited by applicant]
US 20230410384A1 · Pajouh · 2023 [cited by examiner]
WO 2023099233A1 · 2023 [cited by applicant]
Aittala, “Inverse lighting and photorealistic rendering for augmented reality,” Visual Computer, 26:669-678 (2010). [cited by applicant]
Anonymous, “ETSI TR 126 928 V17.0.0 (May 2002),” Technical Report, 5G; Extended Reality (XR) in 5G 3GPP TR 26.928 version 17.0.0 Release 17) (2022) (133 pages). [cited by applicant]
Anonymous, “Get the lighting right,” Platform-Specific Guides, Google ARCore Documentation (2023) (13 pages). [cited by applicant]
Bross et al., “Overview of the Versatile Video Coding (VVC) Standard and its Applications,” IEEE Transactions on Circuits and Systems for Video Technology, 31(10):3736-3764 (2021). [cited by applicant]
Lai et al., “Furion: Engineering High-Quality Immersive Virtual Reality on Today's Mobile Devices,” Paper Session VIII: Frameworks and Such, MobiCom'17, Oct. 16-20, 2017 (13 pages). [cited by applicant]
Ohlsson et al., “Reducing mobility interruption time in 5G networks,” Ericsson Blog, Apr. 2, 2020 (9 pages). [cited by applicant]
Qualcomm Technologies, Inc., “Boundless XR and cloud gaming over 5G,” QCCOMResearch (2021) (33 pages). [cited by applicant]
Rehman et al., “A Survey of Handover Management in Mobile HetNets: Current Challenges and Future Directions,” Applied Sciences, 13(5):3367 (2023). [cited by applicant]
Siriwardhana et al., “A Survey on Mobile Augmented Reality With 5G Mobile Edge Computing: Architectures, Applications, and Technical Aspects,” IEEE Communications Surveys & Tutorials, 23(2):1160-1192, Secondquarter 2021. [cited by applicant]
Wang et al., “Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting,” ICCV 2021 open access, Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 12538-12547 (10 pages… [cited by applicant]
Xia, “New advances for haptic rendering: state of the art,” The Visual Computer 34.2, 271-287 (2018). [cited by applicant]
“Common Test Conditions for MPEG Immersive Video”, ISO/IEC JTC 1/SC29/WG04, N0203, Apr. 29, 2022. [cited by applicant]
“IEEE SA Standards Association—IEEE SA Beyond Standards”, retrieved at https://standards.ieee.org/beyond-standards/, on Jun. 20, 2023. [cited by applicant]
“Immersion | Author Archives: Chris Ullrich”, retrieved at https://www.immersion.com/author/cullrich/, on Jun. 20, 2023. [cited by applicant]
“The Haptics Industry Forum”, retrieved at https://hapticsif.org/, Jun. 20, 2023. [cited by applicant]
Bastian, Matthias , “OpenXR aims to standardize ‘advanced haptics’ for VR and AR”, retrieved at https://mixed-news.com/en/openxr-aims-to-standardize-advanced-haptics-for-vr-and-ar/, Mar. 12, 2022. [cited by applicant]
Evans, Alun , et al., “A pipeline for the creation of progressively rendered web 3D scenes”, Multimedia Tools and Applications, Jan. 28, 2017. [cited by applicant]
Kroon, Bart , “Call for MPEG immersive video test materials”, retrieved at https://mpeg-miv.org/index.php/2022/05/11/call-for-mpeg-immersive-video-test-materials/, May 11, 2022. [cited by applicant]
Lavoue, Guillaume , et al., “Streaming Compressed 3D Data on the Web using JavaScript and WebGL”, In: Proceedings of the 18th international conference on 3D web technology. ACM, 2013, 19-27. [cited by applicant]
Salisbury, Kenneth , et al., “Haptic Rendering: Introductory Concepts”, IEEE Computer Graphics and Applications, 2004, 24-32. [cited by applicant]
Verschoor, M. , et al., “Tactile Rendering Based on Skin Stress Optimization”, ACM Trans. Graph., vol. 39, No. 4, Jan. 13, 2020. [cited by applicant]
Zampoglou, Markos , et al., “Adaptive streaming of complex Web 3D scenes based on the MPEG-DASH standard”, Multimedia Tools and Applications, 77.1, 2016, 125-148. [cited by applicant]
U.S. Appl. No. 18/214,279, filed Jun. 26, 2023, Tatu V.J. Harviainen. [cited by applicant]
U.S. Appl. No. 18/214,283, filed Jun. 26, 2023, Tatu V.J. Harviainen. [cited by applicant]
U.S. Appl. No. 18/518,017, filed Nov. 22, 2023, Tatu V.J. Harviainen. [cited by applicant]
U.S. Appl. No. 18/518,023, filed Nov. 22, 2023, Tatu V.J. Harviainen. [cited by applicant]
Anonymous, “ARcore”, [Retrieved from https://developers.google.com/ar] (8 Pages). [cited by applicant]
Anonymous, “Augmented Reality-More to explore with ARKit 6”, (Apple Inc) [Retrieved from: https://developer. apple.com/augmented-reality/arkit/] (4 pages). [cited by applicant]
Baroffio, Luca, et al., “Coding local and global binary visual features extracted from video sequences”, IEEE Transactions on Image Processing, vol. 24, No. 11, pp. 3546-3560, Nov. 2015, (https://ieeexplore.ieee.org/abs… [cited by applicant]
Li, Guangqiang, et al., “A deep-learning real-time visual SLAM system based on multi-task feature extraction network and self-supervised feature points”, (Jan. 2021) (https://www.sciencedirect.com/science/article/abs/pi… [cited by applicant]
Ma, Jiayi, et al., “Image Matching from Handcrafted to Deep Features: A Survey”, Int J Comput Vis 129, 23-79 (2021). https://doi.org/10.1007/s11263-020-01359-2 (167 pages). [cited by applicant]
Mehrabi, Abbas, et al., “Multi-Tier CloudVR: Leveraging Edge Computing in Remote Rendered Virtual Reality”, ACM Transactions on Multimedia Computing, Communications, and Applications vol. 17 Issue 2 Article No. 49 pp. 1… [cited by applicant]
Qu, Q., et al., “A Comprehensive Review of Machine Learning in Multi-objective Optimization”, (2021 IEEE 4th International Conference on Big Data and Artificial Intelligence) (BDAI), (Qingdao, China), (2021, pp. 7-14). [cited by applicant]
E. Gatti., et al., “Haptic Rules! Augmenting the gaming experience in the traditional games: The case of Foosball,” Jun. 1, 2017 IEEE World Haptics Conference, 432-435 (2017). [cited by applicant]
J. Sachs, et al., “Adaptive 5G Low-Latency Communication for Tactile Internet Services,” Proceedings of the IEEE, 107(2):354-349 (2018). [cited by applicant]
J.-M. Lim et al., “An Audio-Haptic Feedbacks for Enhancing User Experience,” IEEE international Conference on Consumer Electronics, 49-50 (2013). [cited by applicant]
M. Gaudina et al., “Virtual Social Multimedia Streaming with a Novel Haptic Device,” Proceedings of the 35th International Convention MIPRO, 1009-1016 (2012). [cited by applicant]
Cited By (5)
US 12,413,795 US 12,436,617 US 12,574,430 US 12,593,099 US 12,701,271