Granted Patent B2

US 12,333,762 · App. 18/466,426 · Granted Jun 17, 2025

Inventors: Tianheng Wang (San Diego, CA); Stergios Roumeliotis (Los Altos Hills, CA); Shuntaro Yamazaki (Cupertino, CA); Oleg Naroditsky (San Francisco, CA)

Assignee: Apple Inc.

G06T7/85G02B27/0172H04N13/246G02B2027/0134

Abstract

Claims (95)

1. A device, comprising:

cameras configured to be coupled with a rigid body, wherein at least part of respective fields of view of the cameras overlap with one another;

one or more processors; and

a memory, storing program instructions that when executed using the one or more processors, cause the one or more processors to:

perform a calibration process for the cameras, wherein to perform the camera calibration process the program instructions cause the one or more processors to:

determine points of interest within a first set of images captured by the cameras at a first moment in time;

identify pixel coordinate positions of the points of interest of the first set of images;

estimate, via an optimization technique, relative rotational parameters of the cameras, relative to one another, based, at least in part, on the pixel coordinate positions of the points of interest within the captured images at the first moment in time; and

repeat, for a plurality of additional sets of images captured by the cameras at a plurality of additional moments in time, the determination of points of interest, the identification of pixel coordinate positions of the points of interest, and the estimation, via the optimization technique, of the relative rotational parameters of the cameras; and

determine output relative rotational parameters of the calibration process based, at least in part, on a statistical analysis of the estimated relative rotational parameters estimated for the first moment in time and the plurality of additional moments in time; and

provide results of the calibration process, wherein the results comprise the output relative rotational parameters.

2. The device of claim 1 , wherein the optimization technique is a bundle adjustment optimization technique.

3. The device of claim 1 , wherein the first moment in time and the plurality of additional moments in time are asynchronous with respect to one another.

4. The device of claim 1 , wherein at least one or more of the points of interest within the first set of images and the points of interest within the plurality of additional sets of images are different with respect to one another.

5. The device of claim 1 , wherein:

a duration of the calibration process is determined based, at least in part, on at least one of:

a level of current power consumption of the device; or

a level of current processing power of the device; and

the duration of the calibration process comprises the first moment in time and the plurality of additional moments in time.

6. The device of claim 1 , wherein:

a duration of the calibration process is determined based, at least in part, on one or more stop criteria, wherein the stop criteria comprise a convergence threshold to which the relative rotational parameters of the first moment in time and the plurality of additional moments in time converge within; and

the duration of the calibration process comprises the first moment in time and the plurality of additional moments in time.

7. The device of claim 1 , wherein the estimate, via the optimization technique, the rotational parameters of the cameras is additionally based, at least in part, on one or more of:

translational parameters of the cameras relative to the rigid body of the device;

focal lengths of the cameras; or

principal points of the cameras.

8. The device of claim 1 , wherein the device is a head-mounted display (HMD) device.

9. A method, comprising:

performing a calibration process for cameras configured to be coupled with a rigid body of a device, wherein:

at least part of respective fields of view of the cameras overlap with one another; and

the calibration process comprises:

determining points of interest within a first set of images captured by the cameras at a first moment in time;

identifying pixel coordinate positions of the points of interest of the first set of images;

estimating, via an optimization technique, relative rotational parameters of the cameras, relative to one another, based, at least in part, on the pixel coordinate positions of the points of interest within the captured images at the first moment in time; and

repeating, for a plurality of additional sets of images captured by the cameras at a plurality of additional moments in time, the determining the points of interest, the identifying the pixel coordinate positions of the points of interest, and the estimating, via the optimization technique, the relative rotational parameters of the cameras; and

determining output relative rotational parameters of the calibration process based, at least in part, on a statistical analysis of the estimated relative rotational parameters estimated for the first moment in time and the plurality of additional moments in time; and

providing results of the calibration process, wherein the results comprise the output relative rotational parameters.

10. The method of claim 9 , wherein:

the points of interest within the first set of images are determined based, at least in part, on one or more of:

a quality of lighting in the first set of captured images;

respective distances between the points of interest within the first set of images in 3D space and the cameras; or

a spatial distribution within the overlapping fields of view of the cameras of the points of interest within the first set of images; and

the points of interest within the additional sets of images are determined based, at least in part, on one or more of:

a quality of lighting in the additional sets of captured images;

respective distances between the points of interest within the additional sets of images in 3D space and the cameras; or

a spatial distribution within the overlapping fields of view of the cameras of the points of interest within the additional sets of images.

11. The method of claim 9 , wherein:

said determining the points of interest comprises:

identifying respective ones of the points of interest within the first set of images that have high error values; and

removing the respective ones of the points of interest within the first set of images that have the high error values; and

said repeating, for the plurality of additional sets of images, the determining the points of interest comprises:

identifying respective ones of the points of interest within the additional sets of images that have high error values; and

removing the respective ones of the points of interest within the additional sets of images that have the high error values.

12. The method of claim 9 , further comprising:

performing, responsive to said providing the results of the calibration process, a camera calibration verification process for the cameras, wherein the camera calibration verification process comprises:

determining other points of interest within images captured by the cameras at other moments in time;

calculating reprojection error values for the other points of interest based, at least in part, on the output relative rotational parameters; and

determining a camera calibration confidence metric based, at least in part, on the reprojection error values; and

verifying the results of the calibration process based, at least in part, on the camera calibration confidence metric.

13. The method of claim 9 , further comprising:

performing, responsive to said providing the results of the calibration process, a camera calibration verification process for the cameras, wherein the camera calibration verification process comprises:

determining other points of interest within images captured by the cameras at other moments in time;

calculating reprojection error values for the other points of interest based, at least in part, on the output relative rotational parameters; and

determining a camera calibration confidence metric based, at least in part, on the reprojection error values; and

providing a recommendation to perform at least one other calibration process based, at least in part, on the camera calibration confidence metric.

14. The method of claim 13 , wherein the at least one other calibration process pertains to calibration of one or more of:

translational parameters of the cameras relative to one another;

focal lengths of the cameras; or

principal points of the cameras.

15. The method of claim 9 , further comprising:

determining that one or more of the output relative rotational parameters has changed more than a threshold amount with respect to stored relative rotational parameters of the cameras; and

sending the one or more of the output relative rotational parameters to one or more systems of the device that use the cameras.

16. A non-transitory computer-readable medium storing program instructions that, when executed using one or more processors, cause the one or more processors to:

perform a calibration process for cameras, wherein:

at least part of respective fields of view of the cameras overlap with one another; and

the calibration process comprises:

determine points of interest within a first set of images captured by the cameras at a first moment in time;

identify pixel coordinate positions of the points of interest of the first set of images;

estimate, via an optimization technique, relative rotational parameters of the cameras, relative to one another, based, at least in part, on the pixel coordinate positions of the points of interest within the captured images at the first moment in time; and

repeat, for a plurality of additional sets of images captured by the cameras at a plurality of additional moments in time, the determine the points of interest, the identify the pixel coordinate positions of the points of interest, and the estimate, via the optimization technique, the relative rotational parameters of the cameras; and

determine output relative rotational parameters of the calibration process based, at least in part, on a statistical analysis of the estimated rotational parameters estimated for the first moment in time and the plurality of additional moments in time; and

provide results of the calibration process, wherein the results comprise the output relative rotational parameters.

17. The non-transitory computer-readable medium of claim 16 , wherein the program instructions, when executed using the one or more processors, further cause the one or more processors to:

determine that one or more of the output relative rotational parameters have changed more than a threshold amount with respect to stored relative rotational parameters of the cameras; and

send the one or more of the output relative rotational parameters to applications that use the cameras.

18. The non-transitory computer-readable medium of claim 17 , wherein to determine that the one or more of the output relative rotational parameters have changed more than the threshold amount, the program instructions cause the one or more processors to:

perform a camera calibration verification process for the cameras, wherein the camera calibration verification process comprises:

determine other points of interest within images captured by the cameras at other moments in time;

calculate reprojection error values for the other points of interest based, at least in part, on the output relative rotational parameters; and

determine a camera calibration confidence metric based, at least in part, on the reprojection error values; and

verify the results of the calibration process based, at least in part, on the camera calibration confidence metric.

19. The non-transitory computer-readable medium of claim 16 , wherein:

a duration of the calibration process is determined based, at least in part, on one or more stop criteria, wherein the stop criteria comprise a convergence threshold to which the relative rotational parameters pertaining to the first moment in time and the plurality of additional moments in time converge within; and

the duration of the calibration process comprises the first moment in time and the plurality of additional moments in time.

20. The non-transitory computer-readable medium of claim 16 , wherein at least one or more of the points of interest within the first set of images and the points of interest within the plurality of additional sets of images are different with respect to one another.

Assignments (1)

Continuity (2)

References Cited (34)