Granted Patent B2

US 12,406,397 · App. 18/245,816 · Granted Sep 2, 2025

Inventors: Mohamed Souiai (Plantation, FL); Ankur Gupta (Plantation, FL); Igor Napolskikh (Plantation, FL)

Assignee: Magic Leap, Inc.

G06T7/85G06T19/00G06T2207/10012G06T2207/30244

Abstract

Claims (82)

1. A computer-implemented method comprising:

maintaining, in memory:

a three-dimensional model of an environment, and

a plurality of image data clusters that each include data generated from images captured by two or more cameras included in a device, wherein the images represent a portion of the environment in which the device was located; and

jointly determining, for a three-dimensional point represented by an image data cluster from the plurality of image data clusters and using (i) previously estimated coordinates for the three-dimensional point, (ii) the image data cluster, (iii) previously estimated calibration data that represents a spatial relationship between the two or more cameras, (iv) newly estimated coordinates for the three-dimensional point, and (v) newly estimated calibration data that represents the spatial relationship between the two or more cameras:

the newly estimated coordinates for the three-dimensional point for an update to the three-dimensional model or a trajectory of the device; and

the newly estimated calibration data that represents the spatial relationship between the two or more cameras.

2. The method of claim 1 , wherein jointly determining the newly estimated coordinates, and the newly estimated calibration data comprises jointly determining, for the three-dimensional point represented by the image data cluster from the plurality of image data clusters:

the newly estimated coordinates;

the newly estimated calibration data; and

either:

i) an updated three-dimensional model or ii) a trajectory of the device in the environment that includes a physical location for the three-dimensional point; or

a) the updated three-dimensional model, and b) the trajectory of the device in the environment that includes a physical location for the three-dimensional point.

3. The method of claim 2 , comprising:

presenting, on a display, content for the environment using i) the updated three-dimensional model or ii) the trajectory of the device in the environment or iii) both.

4. The method of claim 1 , wherein determining the newly estimated coordinates comprises iteratively determining the newly estimated coordinates by, for each of two or more iterations:

determining newly estimated coordinates for the three-dimensional point using previously estimated coordinates for the three-dimensional point, the image data cluster, and previously estimated calibration data that represents a spatial relationship between the two or more cameras;

determining whether a convergence threshold is satisfied; and

upon determining that the convergence threshold is not satisfied during at least one of the two or more iterations:

setting the newly estimated coordinates for the three-dimensional point as the previously estimated coordinates for the three-dimensional point; and

performing a next iteration using the newly estimated coordinates; or

upon determining that the convergence threshold is satisfied during at least one of the two or more iterations, determining to store the newly estimated coordinates for the three-dimensional point.

5. The method of claim 4 , comprising:

determining, for each of one or more other image data clusters from the plurality of image data clusters that each include data for the three-dimensional point, another newly estimated coordinate for the three-dimensional point;

wherein setting the newly estimated coordinates for the three-dimensional point as the previously estimated coordinates for the three-dimensional point comprises:

averaging the newly estimated coordinates for the three-dimensional point and each of the one or more other newly estimated coordinates for the three-dimensional point to determine average estimated coordinates; and

setting the average estimated coordinates for the three-dimensional point as the previously estimated coordinates for the three-dimensional point.

6. The method of claim 1 , comprising:

averaging, for a first image data cluster and a second image data cluster included in the plurality of image data clusters, a first previously estimated coordinate for the first image data cluster and a second previously estimated coordinate for the second image data cluster to determine an averaged previously estimated coordinate; and

averaging first previously estimated calibration data for the first image data cluster and second previously estimated calibration data for the second image data cluster to determine averaged previously estimated calibration data, wherein:

jointly determining the newly estimated coordinates for the three-dimensional point and the newly estimated calibration data using the previously estimated coordinates and the previously estimated calibration data comprises jointly determining the newly estimated coordinates for the three-dimensional point and the newly estimated calibration data using the averaged previously estimated coordinates and the averaged previously estimated calibration data.

7. The method of claim 6 , wherein:

calibration data comprises translation data, rotation data, and an estimated location data; and

averaging first previously estimated calibration data for the first image data cluster and second previously estimated calibration data for the second image data cluster to determine averaged previously estimated calibration data comprises:

averaging first translation data for the first image data cluster with second translation data for the second image data cluster;

averaging first rotation data for the first image data cluster with second rotation data for the second image data cluster; and

determining to skip averaging a first estimated location data for the first image data cluster and second estimated location data for the second image data cluster.

8. The method of claim 6 , wherein:

jointly determining the newly estimated coordinates for the three-dimensional point and the newly estimated calibration data using the averaged previously estimated coordinates comprises, substantially concurrently:

determining first newly estimated coordinates for the first image data cluster using the first image data cluster and a first copy of the average previously estimated coordinates;

determining second newly estimated coordinates for the second image data cluster using the second image data cluster and a second copy of the average previously estimated coordinates;

determining first newly estimated calibration data for the first image data cluster using the first image data cluster and a first copy of the average previously estimated calibration data; and

determining second newly estimated calibration data for the second image data cluster using the second image data cluster and a second copy of the average previously estimated calibration data.

9. The method of any of claim 6 , wherein the first image data cluster and the second image data cluster include image data for adjacent regions in the three-dimensional model.

10. The method of any of claim 6 , wherein the first image data cluster includes image data captured during a first time period and the second image data cluster includes image data captured during a second time period that is adjacent to the first time period.

11. The method of claim 1 , wherein:

determining the newly estimated coordinates or determining the newly estimated calibration data as part of the joint determination comprises:

receiving, from a first proximity operator, a first partial newly estimated value that the first proximity operator determined using a previously estimated partial value, data for an projection of a point for the image data cluster onto the three-dimensional model, a step size parameter, and a visibility matrix;

receiving, from a second proximity operator, a second partial newly estimated value that the second proximity operator determined using the image data cluster and the visibility matrix; and

combining the first partial newly estimated value and the second partial newly estimated value to determine the newly estimated value.

12. The method of claim 1 , comprising:

receiving data for a plurality of images captured by the two or more cameras; and

determining, using the plurality of images, the plurality of image data clusters that each include data for two or more images that depict a same portion of the environment in which the device was located, wherein the two or more images are included in the plurality of images.

13. The method of claim 1 , wherein jointly determining the newly estimated coordinates and the newly estimated calibration data comprises:

providing, to a proximal splitting engine, the previously estimated coordinates for the three-dimensional point, the image data cluster, and the previously estimated calibration data that represents a spatial relationship between the two or more cameras; and

receiving, from the proximal splitting engine, the newly estimated coordinates and the newly estimated calibration data.

14. The method of claim 1 , wherein:

jointly determining the newly estimated coordinates and the newly estimated calibration data comprises:

providing, to a first proximal splitting engine, the previously estimated coordinates for the three-dimensional point, the image data cluster, and the previously estimated calibration data that represents a spatial relationship between the two or more cameras;

receiving, from the first proximal splitting engine, the newly estimated coordinates;

providing, to a second proximal splitting engine, the image data cluster, and the previously estimated calibration data that represents a spatial relationship between the two or more cameras; and

receiving, from the second proximal splitting engine, the newly estimated calibration data.

15. The method of claim 1 , wherein the calibration data identifies, for a pair of cameras in the two or more cameras, a rotation parameter and a translation parameter that represent the spatial relationship between the pair of cameras.

16. The method of claim 1 , wherein the calibration data identifies, for a pair of cameras in the two or more cameras, a location of the camera with respect to the environment.

17. The method of claim 1 , wherein the environment is a physical environment.

18. The method of claim 1 , comprising:

capturing, by each of two or more cameras and substantially concurrently, an image of the environment; and

generating, using the two or more images, each of which was captured by one of the two or more cameras, three-dimensional data that includes, for an object depicted in each of the two or more images, a three-dimensional point for a feature of the object.

19. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations, comprising:

maintaining, in memory:

a three-dimensional model of an environment, and

a plurality of image data clusters that each include data generated from images captured by two or more cameras included in a device, wherein the images represent a portion of the environment in which the device was located; and

jointly determining, for a three-dimensional point represented by an image data cluster from the plurality of image data clusters and using (i) previously estimated coordinates for the three-dimensional point, (ii) the image data cluster, (iii) previously estimated calibration data that represents a spatial relationship between the two or more cameras, (iv) newly estimated coordinates for the three-dimensional point, and (v) newly estimated calibration data that represents the spatial relationship between the two or more cameras:

the newly estimated coordinates for the three-dimensional point for an update to the three-dimensional model or a trajectory of the device; and

the newly estimated calibration data that represents the spatial relationship between the two or more cameras.

20. A system comprising one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations, comprising:

maintaining, in memory:

a three-dimensional model of an environment, and

a plurality of image data clusters that each include data generated from images captured by two or more cameras included in a device, wherein the images represent a portion of the environment in which the device was located; and

jointly determining, for a three-dimensional point represented by an image data cluster from the plurality of image data clusters and using (i) previously estimated coordinates for the three-dimensional point, (ii) the image data cluster, (iii) previously estimated calibration data that represents a spatial relationship between the two or more cameras, (iv) newly estimated coordinates for the three-dimensional point, and (v) newly estimated calibration data that represents the spatial relationship between the two or more cameras:

the newly estimated coordinates for the three-dimensional point for an update to the three-dimensional model or a trajectory of the device; and

the newly estimated calibration data that represents the spatial relationship between the two or more cameras.

Assignments (2)

Continuity (2)

References Cited (17)