IP Library Granted Patent US 11,030,808
Granted Patent B2
US 11,030,808 · App. 15/789,329 · Granted Jun 8, 2021

Generating time-delayed augmented reality content

Inventors: Stephen Prideaux-Ghee (Bristol, GB); Andre Gosselin (Montreal, CA); Neil Potter (Cambridge, GB); Orit Itzhar (Needham, MA); Aakash Chopra (Austin, TX)
Assignee: PTC Inc.
G06T19/006G06T7/75G06T15/20G06T19/20G06T2207/10028G06T2215/16
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,030,808
App. No.
15/789,329
Granted
Jun 8, 2021
Kind
B2
Abstract

An example method includes: obtaining an image of an object captured by a device during relative motion between the object and the device; determining a location of the device relative to the object during image capture based on one or more attributes of the object in the image; storing the image of the object and the location of the device during image capture; mapping a 3D graphical model representing the object to the object in the image based, at least in part, on the location of the device; receiving, at a time subsequent to capture of the image, first data representing an action to be performed for the object in the image; and generating second data for use in rending content on a display device, where the second data is based on the image stored, the location of the device stored, and at least some information from the model.

Claims (116)

1. A method performed by a computing system, comprising:

obtaining an image of an object captured by an image capturing device in a vicinity of the object during relative motion between the object and the image capturing device;

determining a location of the image capturing device relative to the object during image capture based on one or more attributes of the object in the image;

storing, in computer memory, the image of the object and the location of the image capturing device during image capture;

mapping a three-dimensional (3D) graphical model representing the object to the object in the image based, at least in part, on the location of the image capturing device;

incorporating an update into the 3D graphical model after the 3D model has been mapped to the object in the image, the update representing a current state of at least part of the object that is different from the state of the object when the image was captured;

after the image capturing device has moved away from the vicinity of the object and at a time after the update to the 3D graphical model has been incorporated, retrieving the image from the computer memory at a remote location;

receiving, at the remote location and at a time after the image capturing device has moved away from the vicinity of the object, first data indicating selection by a user of the object in the image that was retrieved; and

in response to receiving the first data, generating second data for use in rendering content on a display device at the remote location, the second data comprising information about the object selected and being based on the image stored, the location of the image capturing device stored, and the 3D graphical model containing the update;

wherein the second data represents the image that was captured augmented with the current state of the at least part of the object obtained from the 3D graphical model containing the update.

2. The method of claim 1 , wherein the first data indicates selection of part of the object in the image that was retrieved.

3. The method of claim 1 , wherein the content comprises the image augmented with the current state of the at least part of the object.

4. The method of claim 1 , wherein the 3D graphical model is part of a digital twin for the object that also includes at least some of the information, and

wherein the method further comprises:

receiving the update after the image has been captured;

wherein the second data represents the image from a perspective of the image capturing device that is based on the location of the image capturing device.

5. The method of claim 1 , wherein the 3D graphical model is part of a digital twin for the object that also includes at least some of the information, and

wherein the method further comprises:

receiving the update after the image has been captured;

wherein the update is received from a sensor associated with the object, the sensor providing the update following capture of the image by the image capturing device, the current state comprising real-time content.

6. The method of claim 1 , wherein the 3D graphical model is part of a digital twin for the object that also includes at least some of the information, and

wherein the method further comprises:

receiving the update after the image has been captured;

wherein the update is received in real-time.

7. The method of claim 1 , wherein the image is a frame of video captured by the image capturing device during the relative motion between the object and the image capturing device;

wherein the location comprises a position and an orientation of the image capturing device relative to the object for each of multiple frames of the video; and

wherein the content comprises the video overlaid with 3D graphical model content and real-time content and presented from a perspective of the image capturing device.

8. The method of claim 1 , wherein determining the location comprises:

obtaining a first size of the object shown in the image, the first size being among the one or more attributes;

obtaining a second size of the object from the 3D graphical model; and

comparing the first size to the second size to determine a distance between the device and the object, the distance being part of the location.

9. The method of claim 1 , wherein determining the location comprises:

identifying a feature of the object shown in the image, the feature being among the one or more attributes; and

determining an orientation of the object relative to the image capturing device based on the feature and based on information about the object in the 3D graphical model, the orientation being part of the location.

10. The method of claim 1 , wherein determining the location of the image capturing device comprises updating the location of the image capturing device as relative positions between the object and the image capturing device change; and

wherein mapping the 3D graphical model to the object in the image is performed for updated locations of the image capturing device.

11. The method of claim 1 , wherein mapping the 3D graphical model to the object in the image comprises associating parts of the 3D graphical model to corresponding parts of the object shown in the image, with a remainder of the 3D graphical model representing parts of the object not shown in the image being positioned relative to the parts of the 3D graphical model associated with the parts of the object shown in the image.

12. The method of claim 1 , wherein at least some of the information represents components interior to the object.

13. One or more non-transitory machine-readable storage media storing instructions that are executable by one or more processing devices to perform operations comprising:

obtaining an image of an object captured by an image capturing device in a vicinity of the object during relative motion between the object and the image capturing device;

determining a location of the image capturing device relative to the object during image capture based on one or more attributes of the object in the image;

storing, in computer memory, the image of the object and the location of the image capturing device during image capture;

mapping a three-dimensional (3D) graphical model representing the object to the object in the image based, at least in part, on the location of the image capturing device;

incorporating an update into the 3D graphical model after the 3D model has been mapped to the object in the image, the update representing a current state of at least part of the object that is different from the state of the object when the image was captured;

after the image capturing device has moved away from the vicinity of the object and at a time after the update to the 3D graphical model has been incorporated, retrieving the image from the computer memory at a remote location;

receiving, at the remote location and at a time after the image capturing device has moved away from the vicinity of the object, first data indicating selection by a user of the object in the image that was retrieved; and

in response to receiving the first data, generating second data for use in rendering content on a display device at the remote location, the second data comprising information about the object selected and being based on the image stored, the location of the image capturing device stored, and the 3D graphical model containing the update;

wherein the second data represents the image that was captured augmented with the current state of the at least part of the object obtained from the 3D graphical model containing the update.

14. The one or more non-transitory machine-readable storage media of claim 13 , wherein determining the location comprises:

obtaining a first size of the object shown in the image, the first size being among the one or more attributes;

obtaining a second size of the object from the 3D graphical model; and

comparing the first size to the second size to determine a distance between the device and the object, the distance being part of the location.

15. The one or more non-transitory machine-readable storage media of claim 13 , wherein determining the location comprises:

identifying a feature of the object shown in the image, the feature being among the one or more attributes; and

determining an orientation of the object relative to the image capturing device based on the feature and based on information about the object in the 3D graphical model, the orientation being part of the location.

16. The one or more non-transitory machine-readable storage media of claim 13 , wherein determining the location of the image capturing device comprises updating the location of the image capturing device as relative positions between the object and the image capturing device change; and

wherein mapping the 3D graphical model to the object in the image is performed for updated locations of the image capturing device.

17. The one or more non-transitory machine-readable storage media of claim 13 , wherein the first data indicates selection of part of the object in the image that was retrieved.

18. The one or more non-transitory machine-readable storage media of claim 13 , wherein the content comprises the image augmented with the current state of the at least part of the object.

19. The one or more non-transitory machine-readable storage media of claim 13 , wherein the 3D graphical model is part of a digital twin for the object that also includes at least some of the information, and

wherein the operations comprise:

receiving the update after the image has been captured;

wherein the second data represents the image from a perspective of the image capturing device that is based on the location of the image capturing device.

20. The one or more non-transitory machine-readable storage media of claim 13 , wherein the 3D graphical model is part of a digital twin for the object that also includes at least some of the information, and

wherein the operations comprise:

receiving the update after the image has been captured;

wherein the update is received from a sensor associated with the object, the sensor providing the update following capture of the image by the image capturing device, the current state comprising real-time content.

21. The one or more non-transitory machine-readable storage media of claim 13 , wherein the 3D graphical model is part of a digital twin for the object that also includes at least some of the information, and

wherein the operations comprise:

receiving the update after the image has been captured;

wherein the update is received in real-time.

22. The one or more non-transitory machine-readable storage media of claim 13 , wherein the image is a frame of video captured by the image capturing device during the relative motion between the object and the image capturing device;

wherein the location comprises a position and an orientation of the image capturing device relative to the object for each of multiple frames of the video; and

wherein the content comprises the video overlaid with 3D graphical model content and real-time content and presented from a perspective of the image capturing device.

23. The one or more non-transitory machine-readable storage media of claim 13 , wherein mapping the 3D graphical model to the object in the image comprises associating parts of the 3D graphical model to corresponding parts of the object shown in the image, with a remainder of the 3D graphical model representing parts of the object not shown in the image being positioned relative to the parts of the 3D graphical model associated with parts of the object shown in the image.

24. The one or more non-transitory machine-readable storage media of claim 13 , wherein at least some of the information represents components interior to the object.

25. A system comprising:

one or more non-transitory machine-readable storage media storing instructions that are executable; and

one or more processing devices to execute the instructions to perform operations comprising:

obtaining an image of an object captured by an image capturing device in a vicinity of the object during relative motion between the object and the image capturing device;

determining a location of the image capturing device relative to the object during image capture based on one or more attributes of the object in the image;

storing, in computer memory, the image of the object and the location of the image capturing device during image capture;

mapping a three-dimensional (3D) graphical model representing the object to the object in the image based, at least in part, on the location of the image capturing device;

incorporating an update into the 3D graphical model after the 3D model has been mapped to the object in the image, the update representing a current state of at least part of the object that is different from the state of the object when the image was captured;

after the image capturing device has moved away from the vicinity of the object and at a time after the update to the 3D graphical model has been incorporated, retrieving the image from the computer memory at a remote location;

receiving, at the remote location and at a time after the image capturing device has moved away from the vicinity of the object, first data indicating selection by a user of the object in the image that was retrieved; and

in response to receiving the first data, generating second data for use in rendering content on a display device at the remote location, the second data comprising information about the object selected and being based on the image stored, the location of the image capturing device stored, and the 3D graphical model containing the update;

wherein the second data represents the image that was captured augmented with the current state of the at least part of the object obtained from the 3D graphical model containing the update.

26. The system of claim 25 , wherein determining the location comprises:

obtaining a first size of the object shown in the image, the first size being among the one or more attributes;

obtaining a second size of the object from the 3D graphical model; and

comparing the first size to the second size to determine a distance between the device and the object, the distance being part of the location.

27. The system of claim 25 , wherein determining the location comprises:

identifying a feature of the object shown in the image, the feature being among the one or more attributes; and

determining an orientation of the object relative to the device based on the feature and based on information about the object in the 3D graphical model, the orientation being part of the location.

28. The system of claim 25 , wherein determining the location of the image capturing device comprises updating the location of the image capturing device as relative positions between the object and the image capturing device change; and

wherein mapping the 3D graphical model to the object in the image is performed for updated locations of the image capturing device.

29. The system of claim 25 , wherein the first data indicates selection of part of the object in the image that was retrieved.

30. The system of claim 25 , wherein the content comprises the image augmented with the current state of the at least part of the object.

31. The system of claim 25 , wherein the 3D graphical model is part of a digital twin for the object that also includes at least some of the information, and

wherein the operations comprise:

receiving the update after the image has been captured;

wherein the second data represents the image from a perspective of the image capturing device that is based on the location of the image capturing device.

32. The system of claim 25 , wherein the 3D graphical model is part of a digital twin for the object that also includes at least some of the information, and

wherein the operations comprise:

receiving the update after the image has been captured;

wherein the update is received from a sensor associated with the object, the sensor providing the update following capture of the image by the image capturing device, the current state comprising real-time content.

33. The system of claim 25 , wherein the 3D graphical model is part of a digital twin for the object that also includes at least some of the information, and

wherein the operations comprise:

receiving the update after the image has been captured;

wherein the update is received in real-time.

34. The system of claim 25 , wherein the image is a frame of video captured by the image capturing device during the relative motion between the object and the image capturing device;

wherein the location comprises a position and an orientation of the image capturing device relative to the object for each of multiple frames of the video; and

wherein the content comprises the video overlaid with 3D graphical model content and real-time content and presented from a perspective of the image capturing device.

35. The system of claim 25 , wherein mapping the 3D graphical model to the object in the image comprises associating parts of the 3D graphical model to corresponding parts of the object shown in the image, with a remainder of the 3D graphical model representing parts of the object not shown in the image being positioned relative to the parts of the 3D graphical model associated with the parts of the object shown in the image.

36. The system of claim 25 , wherein at least some of the information represents components interior to the object.

Assignments (2)
SECURITY INTEREST Recorded Sep 13, 2018
From: PTC INC.
To: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 047480/0888 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 24, 2017
From: PRIDEAUX-GHEE, STEPHEN; GOSSELIN, ANDRE; POTTER, NEIL; ITZHAR, ORIT; CHOPRA, AAKASH
To: PTC INC.
Reel/Frame 043940/0331 →
Continuity (1)
Related Publication 20190122435A1 · Apr 25, 2019
Cited By (1)
US 12,283,009