Granted Patent B2

US 11,094,079 · App. 16/114,772 · Granted Aug 17, 2021

Inventor: William Hoff (Vienna, AT)

Assignee: FACEBOOK TECHNOLOGIES, LLC

G06T7/74G06T19/006G06T2207/10028G06T2207/20081G06T2207/20084G06T2207/30244

Abstract

Claims (76)

1. A method comprising:

during a training phase, capturing a first set of sensor data using both an image sensor and a depth sensor of an augmented reality display device, the first set of sensor data including a first plurality of images of a physical object and corresponding depth data indicating an orientation of the augmented reality display device in relation to the physical object for each respective image;

receiving pose data identifying a pose of the physical object depicted in each image of the physical object captured during the training phase;

applying a convolutional neural network to the first set of sensor data, yielding a first set of features representing the physical object;

forming a plurality of training clusters based on the first set of features generated from the first sensor data, each training cluster associated with a respective pose of the physical object;

storing the plurality of training clusters in a training database;

subsequently, capturing a second set of sensor data using both the image sensor and the depth sensor of the augmented reality display device, the second set of sensor data including a second set of images of the physical object and corresponding depth data indicating an orientation of the augmented reality display in relation to the physical object;

applying the convolutional neural network to the second set of sensor data, yielding a second set of features representing the physical object;

determining, based on the second set of features and the plurality of training clusters, that a first image from the second set of images depicts the physical object at a first respective pose associated with a first training cluster;

determining a variance between a current orientation of the augmented reality display device and a previous orientation of the augmented reality display device when the first image was captured;

determining a current pose of the physical object being captured by the image sensor of the augmented reality display device based on the variance and the first respective pose associated with a first training cluster;

receiving a user-selected placement of virtual content to a static location relative to the physical object; and

displaying the virtual content at the static location on a display of the augmented reality device, a pose of the virtual content corresponding to the current pose of the object.

2. The method of claim 1 , wherein determining that the first image from the second set of images depicts the physical object at the first respective pose associated with the first training cluster comprises:

accessing the plurality of training clusters from the training database; and

matching the second set of features with features included in the first training cluster based on geometric constraints.

3. The method of claim 2 , further comprising:

updating the training database based on matching the second set of features with the features included in the first training cluster.

4. The method of claim 2 , wherein the virtual content includes a virtual object that appears attached to the physical object on the display of the augmented reality display device.

5. The method of claim 4 , wherein the virtual object is associated with the physical object.

6. The method of claim 1 , wherein the plurality of training clusters are formed without accessing prior knowledge about the physically object.

7. The method of claim 1 , further comprising:

accessing a computer-aided design (CAD) model of the physical object; and

limiting a collecting of training data based on the CAD model of the physical object wherein the plurality of training clusters are further formed based on limiting the collecting of training data.

8. The method of claim 1 , further comprising:

accessing a device pose data of the augmented reality display device based on a visual inertial system in the augmented reality display device, wherein the orientation of augmented reality display device in relation to the physical object for each respective image is further based on the device pose data.

9. The method of claim 1 , further comprising:

detecting parts of the physical object and positions of components of the physical object based on the first set of features corresponding the physical object; and

identifying a state of the physical object based on the parts of the physical object and positions of components of the physical object.

10. An augmented reality display device comprising:

one or more computer processors; and

one or more computer-readable mediums storing instructions that, when executed by the one or more computer processors, cause the augmented reality display device to perform operations comprising:

during a training phase, capturing a first set of sensor data using both an image sensor and a depth sensor of the augmented reality display device, the first set of sensor data including a first plurality of images of a physical object and corresponding depth data indicating an orientation of the augmented reality display device in relation to the physical object for each respective image;

receiving pose data identifying a pose of the physical object depicted in each image of the physical object captured during the training phase;

applying a convolutional neural network to the first set of sensor data, yielding a first set of features representing the physical object;

forming a plurality of training clusters based on the first set of features generated from the first sensor data, each training cluster associated with a respective pose of the physical object;

storing the plurality of training clusters in a training database;

subsequently, capturing a second set of sensor data using both the image sensor and the depth sensor of the augmented reality display device, the second set of sensor data including a second set of images of the physical object and corresponding depth data indicating an orientation of the augmented reality display in relation to the physical object;

applying the convolutional neural network to the second set of sensor data, yielding a second set of features representing the physical object;

determining, based on the second set of features and the plurality of training clusters, that a first image from the second set of images depicts the physical object at a first respective pose associated with a first training cluster;

determining a variance between a current orientation of the augmented reality display device and a previous orientation of the augmented reality display device when the first image was captured;

determining a current pose of the physical object being captured by the image sensor of the augmented reality display device based on the variance and the first respective pose associated with a first training cluster;

receiving a user-selected placement of virtual content to a static location relative to the physical object; and

displaying the virtual content at the static location on a display of the augmented reality device, a pose of the virtual content corresponding to the current pose of the object.

11. The augmented reality display device of claim 10 , wherein determining that the first image from the second set of images depicts the physical object at the first respective pose associated with the first training cluster comprises:

accessing the plurality of training clusters from the training database; and

matching the second set of features with features included in the first training cluster based on geometric constraints.

12. The augmented reality display device of claim 11 , the operations further comprising:

updating the training database based on matching the second set of features with the features included in the first training cluster.

13. The augmented reality display device of claim 11 , wherein the virtual content includes a virtual object that appears attached to the physical object on the display of the augmented reality display device.

14. The augmented reality display device of claim 13 , wherein the virtual object is associated with the physical object.

15. The augmented reality display device of claim 10 , wherein the plurality of training clusters are formed without accessing prior knowledge about the physically object.

16. The augmented reality display device of claim 10 , the operations further comprising:

accessing a computer-aided design (CAD) model of the physical object; and

limiting a collecting of training data based on the CAD model of the physical object wherein the plurality of training clusters are further formed based on limiting the collecting of training data.

17. The augmented reality display device of claim 10 , the operations further comprising:

accessing a device pose data of the augmented reality display device based on a visual inertial system in the augmented reality display device, wherein the orientation of augmented reality display device in relation to the physical object for each respective image is further based on the device pose data.

18. The augmented reality display device of claim 10 , the operations further comprising:

detecting parts of the physical object and positions of components of the physical object based on the first set of features corresponding the physical object; and

identifying a state of the physical object based on the parts of the physical object and positions of components of the physical object.

19. A non-transitory computer-readable medium storing instructions that, when executed by one or more computer processors of an augmented reality display device, cause the augmented reality display device to perform operations comprising:

during a training phase, capturing a first set of sensor data using both an image sensor and a depth sensor of the augmented reality display device, the first set of sensor data including a first plurality of images of a physical object and corresponding depth data indicating an orientation of the augmented reality display device in relation to the physical object for each respective image;

receiving pose data identifying a pose of the physical object depicted in each image of the physical object captured during the training phase;

applying a convolutional neural network to the first set of sensor data, yielding a first set of features representing the physical object;

forming a plurality of training clusters based on the first set of features generated from the first sensor data, each training cluster associated with a respective pose of the physical object;

storing the plurality of training clusters in a training database;

subsequently, capturing a second set of sensor data using both the image sensor and the depth sensor of the augmented reality display device, the second set of sensor data including a second set of images of the physical object and corresponding depth data indicating an orientation of the augmented reality display in relation to the physical object;

applying the convolutional neural network to the second set of sensor data, yielding a second set of features representing the physical object;

determining, based on the second set of features and the plurality of training clusters, that first image from the second set of images depicts the physical object at a first respective pose associated with a first training cluster;

determining a variance between a current orientation of the augmented reality display device and a previous orientation of the augmented reality display device when the first image was captured;

determining a current pose of the physical object being captured by the image sensor of the augmented reality display device based on the variance and the first respective pose associated with a first training cluster;

receiving a user-selected placement of virtual content to a static location relative to the physical object; and

displaying the virtual content at the static location on a display of the augmented reality device, a pose of the virtual content corresponding to the current pose of the object.

20. The non-transitory computer-readable medium of claim 19 , wherein determining that the first image from the second set of images depicts the physical object at the first respective pose associated with the first training cluster comprises:

accessing the plurality of training clusters from the training database; and

matching the second set of features with features included in the first training cluster based on geometric constraints.

Assignments (11)

Continuity (1)

Cited By (1)

US 12,260,634