IP Library Granted Patent US 10,665,028
Granted Patent B2
US 10,665,028 · App. 16/100,126 · Granted May 26, 2020

Mobile persistent augmented-reality experiences

Inventors: Matthew Adam Simari (Seattle, WA); Alvaro Collet Romea (Seattle, WA); Krishnan Kumar Ramnath (Redmond, WA)
Assignee: Facebook, Inc.
G06T19/006G06T7/75G06T19/20G06Q50/01
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,665,028
App. No.
16/100,126
Granted
May 26, 2020
Kind
B2
Abstract

In one embodiment, a method includes determining, using one or more location sensors of a computing device, an approximate location of the computing device, identifying a content object located within a threshold distance of the approximate location, wherein an augmented-reality map associates the content object with a stored model of a real-world object and specifies a location of the content object on or relative to the stored model of the real-world object, obtaining an image from a camera of the device, identifying, in the image, a target real-world object that matches the stored model of the real-world object, determining a content object location based on a location of the target real-world object and the location of the content object on or relative to the model of the real-world object, and displaying the content object at the content object location.

Claims (55)

1. A method comprising:

by a computing device, determining, using one or more location sensors of the computing device, an approximate location of the computing device;

by the computing device, accessing an augmented-reality map associated with a geographic area that comprises the approximate location of the computing device, wherein the augmented-reality map specifies that the geographic area contains one or more predetermined regions defined by corresponding three-dimensional models;

by the computing device, identifying, using the augmented-reality map, a real-world object located in at least one of the predetermined regions;

by the computing device, identifying, using the augmented-reality map, a content object associated with the real-world object, wherein the augmented-reality map associates the content object with a stored model of the real-world object and a stored model of the content object, and specifies a location of the stored model of the content object relative to the stored model of the real-world object;

by the computing device, obtaining an image from a camera of the computing device;

by the computing device, identifying, in the image, a target real-world object that matches the stored model of the real-world object;

by the computing device, determining, using a tracking algorithm, a location of the real-world object relative to the computing device;

by the computing device, determining a content object location based on the location of the target real-world object relative to the computing device and the location of the stored model of the content object relative to the stored model of the real-world object; and

by the computing device, displaying the content object at the content object location.

2. The method of claim 1 , wherein the augmented-reality map associates the one or more content objects with coordinates of locations within the one or more predetermined regions defined by corresponding three-dimensional models.

3. The method of claim 1 , wherein the one or more predetermined regions comprise one or more rooms in one or more buildings.

4. The method of claim 1 , wherein identifying, using the augmented-reality map, a content object associated with the real-world object comprises searching the augmented-reality map for one or more content objects having associated locations within a threshold distance of the approximate location,

wherein the threshold distance is associated with the content object.

5. The method of claim 4 , wherein the threshold distance is based on an accuracy level of the location sensors.

6. The method of claim 1 , wherein the target real-world object matches the stored model of the real-world object when a plurality of target feature points on the target real-world object correspond to a plurality of model feature points associated with the stored model of the real-world object.

7. The method of claim 6 , wherein the target feature points correspond to the model feature points when the target feature points represent the same shape as the model feature points.

8. The method of claim 6 , wherein identifying, in the image, a target real-world object that matches the model of the real-world object comprises:

identifying candidate feature points in the image, wherein the candidate feature points are on edges or corners in the image;

identifying a plurality of the candidate feature points that correspond to the model feature points, wherein the target feature points on the target real-world object comprise the identified plurality of the candidate feature points.

9. The method of claim 1 , wherein displaying the content object comprises rendering the content object based on the stored model of the content object.

10. The method of claim 9 , wherein displaying the content object comprises superimposing the rendering of the content object over the image at a location in the image based on the location of the target real-world object.

11. The method of claim 1 , wherein the location of the stored model of the content object associated with the content object is a location in three-dimensional space and is on a surface of the stored model of the real-world object.

12. The method of claim 1 , wherein the determining the content object location comprises:

by the computing device, retrieving, from the augmented-reality map, a content object offset associated with the content object and further associated with the stored model of the real-world object,

wherein the content object offset specifies the location of the content object relative to the stored model of the real-world object, and the content object location is based on a sum of the location of the target real-world object and the content object offset.

13. The method of claim 12 , wherein the content object offset comprises a three-dimensional vector, and the method further comprises:

by the computing device, transforming the three-dimensional vector based on an orientation of the target real-world object.

14. The method of claim 1 , wherein the content object comprises text content, image content, animation content, video content, or a combination thereof.

15. The method of claim 1 , further comprising:

by the computing device, receiving a new content object and a name that identifies a real-world object; and

by the computing device, storing the new content object in the augmented-reality map in association with a stored model of the real-world object, wherein the stored model of the real-world object is identified based on the name.

16. The method of claim 1 , wherein the augmented-reality map is stored on a social-networking system server.

17. A system comprising: one or more processors; and a memory coupled to the processors comprising instructions executable by the processors, the processors being operable when executing the instructions to:

determine, using one or more location sensors of a computing device, an approximate location of the computing device;

access an augmented-reality map associated with a geographic area that comprises the approximate location of the computing device, wherein the augmented-reality map specifies that the geographic area contains one or more predetermined regions defined by corresponding three-dimensional models;

identify, using the augmented-reality map, a real-world object located in at least one of the predetermined regions;

identify, using the augmented-reality map, a content object associated with the real-world object, wherein the augmented-reality map associates the content object with a stored model of the real-world object and a stored model of the content object, and specifies a location of the stored model of the content object relative to the stored model of the real-world object;

obtain an image from a camera of the computing device;

identify, in the image, a target real-world object that matches the stored model of the real-world object;

determine, using a tracking algorithm, a location of the real-world object relative to the computing device;

determine a content object location based on the location of the target real-world object relative to the computing device and the location of the stored model of the content object relative to the stored model of the real-world object; and

display the content object at the content object location.

18. The system of claim 17 , wherein the augmented-reality map associates the one or more content objects with coordinates of locations within the one or more predetermined regions defined by corresponding three-dimensional models.

19. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

determine, using one or more location sensors of a computing device, an approximate location of the computing device;

access an augmented-reality map associated with a geographic area that comprises the approximate location of the computing device, wherein the augmented-reality map specifies that the geographic area contains one or more predetermined regions defined by corresponding three-dimensional models;

identify, using the augmented-reality map, a real-world object located in at least one of the predetermined regions;

identify, using the augmented-reality map, a content object associated with the real-world object, wherein the augmented-reality map associates the content object with a stored model of the real-world object and a stored model of the content object, and specifies a location of the stored model of the content object relative to the stored model of the real-world object;

obtain an image from a camera of the computing device;

identify, in the image, a target real-world object that matches the stored model of the real-world object;

determine, using a tracking algorithm, a location of the real-world object relative to the computing device;

determine a content object location based on the location of the target real-world object relative to the computing device and the location of the stored model of the content object relative to the stored model of the real-world object; and

display the content object at the content object location.

20. The media of claim 19 , wherein the map associates the one or more content objects with coordinates of locations within the one or more predetermined regions defined by corresponding three-dimensional models.

Assignments (2)
CHANGE OF NAME Recorded Dec 20, 2021
From: FACEBOOK, INC.
To: META PLATFORMS, INC.
Reel/Frame 058553/0802 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 11, 2020
From: SIMARI, MATTHEW ADAM; COLLET ROMEA, ALVARO; RAMNATH, KRISHNAN KUMAR
To: FACEBOOK, INC.
Reel/Frame 051784/0276 →