IP Library › Granted Patent US 12,205,233
Granted Patent B2
US 12,205,233 · App. 18/663,627 · Granted Jan 21, 2025

Systems and methods for dynamic interaction with an augmented reality environment

Inventors: Alexander Mascarin (Kleinburg, CA); Stephan Peralta (Etobicoke, CA); Andrea Tuzi (Innisfil, CA); Matthew David Presta (Vaughan, CA); Michael James Presta (Nobleton, CA)
Assignee: EQPME INC.
G06T19/006G06T7/13
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,205,233
App. No.
18/663,627
Granted
Jan 21, 2025
Kind
B2
Abstract

Methods and systems are provided for dynamic interaction with an augmented reality environment. In some embodiments, the systems and methods are directed at dynamically interacting with machinery within the augmented reality environment via an augmented reality device. The method involves analyzing a site in which the machinery is to be operated to capture environment data; displaying on the augmented reality device an augmented reality representation of the machinery as an overlay of a portion of an augmented reality environment; receiving an interaction request for interacting with the augmented reality representation of the machinery; determining whether the interaction request can be completed in respect of the machinery within the site; and in response to determining that the interaction request can be completed, displaying the augmented reality representation of the machinery in accordance with the interaction request, otherwise, indicating that the interaction request cannot be completed within the site.

Claims (63)

1. A method for dynamically interacting with an augmented reality environment, the method comprising:

capturing, at an augmented reality device, environment data for estimating a site boundary of a site;

receiving a connection request via a network from a remote device to access the augmented reality device, comprising:

generating a remote connection invitation to the remote device to access the augmented reality device; and

receiving the connection request in response to the remote connection invitation;

authenticating the connection request to determine whether to grant access to the augmented reality device;

in response to successfully authenticating the connection request, receiving an interaction request via the remote device for interacting with an augmented reality representation of an object;

determining, based on the captured environment data, whether the interaction request can be completed in respect of the object within the site; and

displaying the augmented reality representation of the object in accordance with the interaction request.

2. The method of claim 1 wherein the connection request comprises a request to access one or more augmented reality devices, and in response to successfully authenticating the connection request to at least two augmented reality devices, synchronizing the interaction request received from the remote device between the at least two augmented reality devices.

3. The method of claim 1 , wherein the interaction request comprises a support input for facilitating a local interaction request received at the augmented reality device.

4. The method of claim 3 , wherein the local interaction request comprises a transaction request in respect of the object.

5. The method of claim 1 further comprises receiving a remote connection request from the augmented reality device to request connection with the remote device to initiate an interaction request in respect of the object from the remote device.

6. The method of claim 1 further comprises offering a recommendation for the interaction request to the remote device when the interaction request cannot be completed within the site.

7. The method of claim 1 , wherein:

determining whether the interaction request can be completed for the object within the site comprises determining whether an operating range of the object for completing the interaction request is restricted by one or more of the site boundary and an obstacle within the site.

8. The method of claim 7 further comprises:

in response to determining that the interaction request cannot be completed, recommending an alternative object suitable for the site.

9. The method of claim 1 further comprises displaying on the augmented reality device at least one cause preventing completion of the interaction request when the interaction request cannot be completed within the site.

10. The method of claim 9 further comprises receiving an override input from one of the augmented reality device and the remote device to initiate completion of the interaction request and to disregard the at least one cause preventing completion of the interaction request.

11. The method of claim 1 , wherein the interaction request comprises an engagement with one or more components of the object.

12. The method of claim 11 further comprises, in response to determining that the interaction request cannot be completed:

indicating the one or more components of the object is unsuitable for the site; and

recommending an alternative component compatible for the object and suitable for the site.

13. The method of claim 1 further comprises:

determining an available space within the augmented reality environment for displaying one or more advertising content without interfering with the interaction request; and

displaying the one or more advertising content within the available space.

14. A method for dynamically interacting with an augmented reality environment, the method comprising:

capturing, at an augmented reality device, environment data for estimating a site boundary of a site;

receiving a connection request via a network from a remote device to access the augmented reality device;

authenticating the connection request to determine whether to grant access to the augmented reality device;

mirroring a display of the remote device with a display of the augmented reality device;

in response to successfully authenticating the connection request, receiving an interaction request via the remote device for interacting with an augmented reality representation of an object;

determining, based on the captured environment data, whether the interaction request can be completed in respect of the object within the site; and

displaying the augmented reality representation of the object in accordance with the interaction request.

15. The method of claim 14 wherein the connection request comprises a request to access one or more augmented reality devices, and in response to successfully authenticating the connection request to at least two augmented reality devices, synchronizing the interaction request received from the remote device between the at least two augmented reality devices.

16. The method of claim 14 , wherein the interaction request comprises a support input for facilitating a local interaction request received at the augmented reality device.

17. A system for dynamically interacting with an augmented reality environment, the system comprising a processor operable to:

capture environment data for estimating a site boundary of a site;

receive a connection request via a network from a remote device to access the augmented reality device, comprising:

generating a remote connection invitation to the remote device to access the augmented reality device; and

receiving the connection request in response to the remote connection invitation;

authenticate the connection request to determine whether to grant a remote device access to the augmented reality device;

in response to successfully authenticating the connection request, receive an interaction request via the remote device for interacting with an augmented reality representation of an object;

determine, based on the captured environmental data, whether the interaction request can be completed in respect of the object within the site; and

display the augmented reality representation of the object in accordance with the interaction request.

18. The system of claim 17 , wherein the processor is operable to mirror a display of the remote device with a display of the augmented reality device.

19. The system of claim 17 wherein the connection request comprises a request to access one or more augmented reality devices, and in response to successfully authenticating the connection request to at least two augmented reality devices, the processor is operable to synchronize the interaction request received from the remote device between the at least two augmented reality devices.

20. The system of claim 17 , wherein the interaction request comprises a support input for facilitating a local interaction request received at the augmented reality device.

21. The system of claim 19 , wherein the local interaction request comprises a transaction request in respect of the object.

22. The system of claim 17 , wherein the processor is operable to receive a remote connection request from the augmented reality device to request connection with the remote device to initiate an interaction request in respect of the object from the remote device.

23. The system of claim 17 , wherein the processor is operable to offer a recommendation for the interaction request to the remote device when the interaction request cannot be completed within the site.

24. The system of claim 17 , wherein the processor is operable to determine whether an operating range of the object for completing the interaction request is restricted by one or more of the site boundary and an obstacle within the site.

25. The system of claim 24 , wherein in response to determining that the interaction request cannot be completed, the processor is operable to recommend an alternative object suitable for the site.

26. The system of claim 17 , wherein the processor is operable to display on the augmented reality device at least one cause preventing completion of the interaction request when the interaction request cannot be completed within the site.

27. The system of claim 17 , wherein the processor is operable to receive an override input from one of the augmented reality device and the remote device to initiate completion of the interaction request and to disregard the at least one cause preventing completion of the interaction request.

28. The system of claim 17 , wherein the interaction request comprises an engagement with one or more components of the object.

29. The system of claim 28 , wherein the processor is operable to:

indicate the one or more components of the object is unsuitable for the site; and

recommend an alternative component compatible for the object and suitable for the site.

30. The system of claim 28 , wherein the processor is operable to:

determine an available space within the augmented reality environment for displaying one or more advertising content without interfering with the interaction request; and

display the one or more advertising content within the available space.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 14, 2024
From: MASCARIN, ALEXANDER; PERALTA, STEPHAN; TUZI, ANDREA; PRESTA, MATTHEW DAVID; PRESTA, MICHAEL JAMES
To: EQPME INC.
Reel/Frame 067409/0691 →
Continuity (5)
Continuation 18229386 · Aug 2, 2023
Continuation In Part 18316498 · May 12, 2023
Continuation 17984338 · Nov 10, 2022
Provisional Application 63394716 · Aug 3, 2022
Related Publication 20240296637A1 · Sep 5, 2024
References Cited (37)
US 5646844A · Gudat et al. · 1997 [cited by applicant]
US 9070216B2 · Golparvar-Fard et al. · 2015 [cited by applicant]
US 9222771B2 · Rosengaus et al. · 2015 [cited by applicant]
US 10580207B2 · Pejic et al. · 2020 [cited by applicant]
US 20080071559A1 · Arrasvuori · 2008 [cited by applicant]
US 20090061901A1 · Arrasvuori et al. · 2009 [cited by applicant]
US 20100185529A1 · Chestnut · 2010 [cited by applicant]
US 20130234850A1 · Lee · 2013 [cited by applicant]
US 20140025481A1 · Kang et al. · 2014 [cited by applicant]
US 20160049005A1 · Mullins et al. · 2016 [cited by applicant]
US 20160165170A1 · McRae · 2016 [cited by applicant]
US 20170116644A1 · Huang et al. · 2017 [cited by applicant]
US 20180349946A1 · Nguyen · 2018 [cited by applicant]
US 20200258144A1 · Chaturvedi et al. · 2020 [cited by applicant]
US 20200402219A1 · Gilbert · 2020 [cited by applicant]
US 20210020141A1 · Yuasa · 2021 [cited by applicant]
US 20220066621A1 · Appelbaum · 2022 [cited by examiner]
CA 3105792A1 · 2021 [cited by applicant]
WO 2018060783A1 · 2018 [cited by applicant]
PIX4D, “Turn your mobile device into a professional 3D scanner”, https://www.pix4d.com/product/pix4dcatch, downloaded May 11, 2022. [cited by applicant]
Sitemark, “Software for solar construction”, https://www.sitemark.com/industries/construction/, downloaded May 11, 2022. [cited by applicant]
ThreeKit, “3D Product Configurator & Augmented Reality for Commerce”, https://www.threekit.com/, downloaded Jun. 28, 2022. [cited by applicant]
JLG, “Product Announcement”, May 2022. [cited by applicant]
Strange, “Now You Can Drive Elon Musk's Tesla Cybertruck in AR from the Comfort of Your Home via iOS & Android”, Dec. 9, 2019, <https://mobile-ar.reality.news/news/now-you-can-drive-elon-musks-tesla-cybertruck-ar-from-c… [cited by applicant]
Makarov, “12 Augmented Reality Trends of 2023: New Milestones in Immersive Technology”, Aug. 2, 2022, <https://mobidev.biz/blog/augmented-reality-trends-future-ar-technologies> retrieved on Jun. 19, 2023. [cited by applicant]
“On-device voice commands in AR Glasses—Vuzix”, <https://vivoka.com/project/vuzix/>, Vivoka, Oct. 2021, retrieved on Jun. 7, 2023. [cited by applicant]
Echo3D, “Top 5 Augmented Reality (AR) Remote Assistance Tools”, <https://medium.com/echo3d/top-5-augmented-reality-ar-remote-assistance-tools-76e830cddf99>, Medium, Jul. 22, 2021, retrieved on Jun. 7, 2023. [cited by applicant]
Virsabi, “Tesla model 3”, <https://virsabi.com/tesla/> (Jul. 15, 2019), retrieved on Jun. 7, 2023. [cited by applicant]
Zero Density, “Augmented Reality—Revolutionize Your Storytelling with Augmented Reality Solutions”, <https://www.zerodensity.tv/solutions/augmented-reality/> (Apr. 6, 2023), retrieved on Jun. 14, 2023. [cited by applicant]
Virsabi, “Metaverse in Business”, <https://virsabi.com/> (Apr. 20, 2017), retrieved on Jun. 14, 2023. [cited by applicant]
Trimble <https://sitevision.trimble.com/> (Oct. 24, 2018), retrieved on Jun. 14, 2023. [cited by applicant]
Mantooth, TJ, “IP Issues for Retail Businesses Advertising in Augmented Reality”, Jul. 26, 2022, <https://ipwatchdog.com/2022/07/26/ip-issues-retail-businesses-advertising-augmented-reality/id=150383/> (retrieved on Jul… [cited by applicant]
Youtube. “Tesla Cybertruck doesn't fit in the average garage!”, Dec. 6, 2019, https://youtu.be/PuZz6OFMW5A (retrieved on Jul. 31, 2023). [cited by applicant]
AR Cybertruck. LuvBug Media, Mar. 8, 2020. Google Play Store, <https://play.google.com/store/apps/details?id=com.luvbug.cybertruck&pli=1> (retrieved on on Jul. 31, 2023). [cited by applicant]
International Search Report and Written Opinion mailed Oct. 25, 2023 in PCT/CA2023/051032. [cited by applicant]
Mourtzis et al., “Real-Time Remote Maintenance Support Based on Augmented Reality (AR)”, 2020. (Year: 2020). [cited by applicant]
Zhao et al., “Voice Interaction for Augmented Reality Navigation Interfaces with Natural Language Understanding”, IEEE, 2020. (Year: 2020). [cited by applicant]