BUNDLE ADJUSTMENT USING EPIPOLAR CONSTRAINTS
Methods, systems, and apparatus for performing bundling adjustment using epipolar constraints. A method includes receiving image data from a headset for a particular pose. The image data includes a first image from a first camera of the headset and a second image from a second camera of the headset. The method includes identifying at least one key point in a three-dimensional model of an environment at least partly represented in the first image and the second image and performing bundle adjustment. Bundle adjustment is performed by jointly optimizing a reprojection error for the at least one key point and an epipolar error for the at least one key point. Results of the bundle adjustment are used to perform at least one of (i) updating the three-dimensional model, (ii) determining a position of the headset at the particular pose, or (iii) determining extrinsic parameters of the first camera and second camera.
1 - 20 . (canceled)
21 . A method comprising:
receiving image data for a pose of a headset, the image data comprising (i) first image data from a first camera of the headset and (ii) second image data from a second camera of the headset;
identifying a set of key points in a three-dimensional model of an environment at least partly represented in the first image data and the second image data;
performing bundle adjustment using the first image data and second image data by jointly optimizing (i) a reprojection error for the key points of the set of key points; (ii) an epipolar error for the key points of the set of key points; and (iii) a factory calibration error, wherein the factory calibration error comprises a change in relative position between the first camera and a reference position on the headset, wherein jointly optimizing comprises:
calculating a total error, wherein calculating the total error comprises summing together:
reprojection errors for the set of key points,
epipolar errors for the set of key points, and
factory calibration errors for the set of key points; and
using results of the bundle adjustment to perform at least one of (i) updating, as an updated three-dimensional model, the three-dimensional model, (ii) determining a position of the headset at the pose, or (iii) determining extrinsic parameters of the first camera and second camera.
22 . The method of claim 21 , wherein the headset comprises an augmented reality display, further comprising displaying an image based on the updated three-dimensional model.
23 . The method of claim 21 , wherein jointly optimizing the error comprises minimizing the total error across each of the set of key points.
24 . The method of claim 21 , wherein updating the three-dimensional model comprises updating positions of one or more key points in the three-dimensional model.
25 . The method of claim 21 , wherein the first image data and the second image data are captured simultaneously.
26 . The method of claim 21 , wherein the extrinsic parameters comprise a translation and rotation that indicate a relationship of the first camera and/or the second camera with respect to a reference position on the headset.
27 . The method of claim 21 , wherein the first image data corresponds to a first pose of the headset and the second image data corresponds to a second pose of the headset, wherein a deformation of the headset occurs between the first pose of the headset and the second pose of the headset,
wherein at least one of the key points represents the deformation of the headset that occurred between the first pose and the second pose of the headset.
28 . The method of claim 21 , further comprising:
receiving additional image data from the first and second cameras at each of a plurality of additional poses along a path of movement of the headset; and
determining additional extrinsic parameters for the first and second cameras for at least some of the plurality of additional poses using results of the jointly optimizing involving the epipolar error.
29 . The method of claim 21 , further comprising:
receiving, from the headset, additional image data for multiple poses of the headset;
identifying at least one second key point in the three-dimensional model of the environment at least partly represented in the additional image data;
performing bundle adjustment for each of the multiple poses by jointly optimizing (i) a reprojection error for the at least one key point based on the additional image data and (ii) an epipolar error for the at least one key point based on the additional image data; and
using results of the bundle adjustment for each of the multiple poses to perform at least one of: (i) updating the three-dimensional model, (ii) determining another position of the headset at each of the multiple poses, or (iii) determining other extrinsic parameters of the first camera and the second camera at each of the multiple poses.
30 . A system comprising one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations including:
receiving image data for a pose of a headset, the image data comprising (i) a first image data from a first camera of the headset and (ii) a second image data from a second camera of the headset;
identifying a set of key points in a three-dimensional model of an environment at least partly represented in the first image data and the second image data;
performing bundle adjustment using the first image data and second image data by jointly optimizing (i) a reprojection error for the key points of the set of key points; (ii) an epipolar error for the key points of the set of key points; and (iii) a factory calibration error, wherein the factory calibration error comprises a change in relative position between the first camera and a reference position on the headset, wherein jointly optimizing comprises:
calculating a total error, wherein calculating the total error comprises summing together:
reprojection errors for the set of key points,
epipolar errors for the set of key points, and
factory calibration errors for the set of key points; and
using results of the bundle adjustment to perform at least one of (i) updating, as an updated three-dimensional model, the three-dimensional model, (ii) determining a position of the headset at the pose, or (iii) determining extrinsic parameters of the first camera and second camera.
31 . The system of claim 30 , wherein the headset comprises an augmented reality display, further comprising displaying an image based on the updated three-dimensional model.
32 . The system of claim 30 , wherein jointly optimizing the error comprises minimizing the total error across each of the set of key points.
33 . The system of claim 30 , wherein updating the three-dimensional model comprises updating positions of one or more key points in the three-dimensional model.
34 . The system of claim 30 , further comprising:
receiving additional image data from the first and second cameras at each of a plurality of additional poses along a path of movement of the headset; and
determining additional extrinsic parameters for the first and second cameras for at least some of the plurality of additional poses using results of the jointly optimizing involving the epipolar error.
35 . The system of claim 30 , wherein the first image data corresponds to a first pose of the headset and the second image data corresponds to a second pose of the headset, wherein a deformation of the headset occurs between the first pose of the headset and the second pose of the headset,
wherein at least one of the key points represents the deformation of the headset that occurred between the first pose and the second pose of the headset.
36 . The system of claim 30 , further comprising:
receiving, from the headset, additional image data for multiple poses of the headset;
identifying at least one second key point in the three-dimensional model of the environment at least partly represented in the additional image data;
performing bundle adjustment for each of the multiple poses by jointly optimizing (i) a reprojection error for the at least one key point based on the additional image data and (ii) an epipolar error for the at least one key point based on the additional image data; and
using results of the bundle adjustment for each of the multiple poses to perform at least one of: (i) updating the three-dimensional model, (ii) determining another position of the headset at each of the multiple poses, or (iii) determining other extrinsic parameters of the first camera and the second camera at each of the multiple poses.
37 . A non-transitory computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations including:
receiving image data for a pose of a headset, the image data comprising (i) a first image data from a first camera of the headset and (ii) a second image data from a second camera of the headset;
identifying a set of key points in a three-dimensional model of an environment at least partly represented in the first image data and the second image data;
performing bundle adjustment using the first image data and second image data by jointly optimizing (i) a reprojection error for the key points of the set of key points; (ii) an epipolar error for the key points of the set of key points; and (iii) a factory calibration error, wherein the factory calibration error comprises a change in relative position between the first camera and a reference position on the headset, wherein jointly optimizing comprises:
calculating a total error, wherein calculating the total error comprises summing together:
reprojection errors for the set of key points,
epipolar errors for the set of key points, and
factory calibration errors for the set of key points; and
using results of the bundle adjustment to perform at least one of (i) updating, as an updated three-dimensional model, the three-dimensional model, (ii) determining a position of the headset at the pose, or (iii) determining extrinsic parameters of the first camera and second camera.
38 . The non-transitory computer storage medium of claim 37 , further comprising providing instructions for displaying an image based on the updated three-dimensional model.
39 . The non-transitory computer storage medium of claim 37 , further comprising:
receiving additional image data from the first and second cameras at each of a plurality of additional poses along a path of movement of the headset; and
determining additional extrinsic parameters for the first and second cameras for at least some of the plurality of additional poses using results of the jointly optimizing involving the epipolar error.
40 . The non-transitory computer storage medium of claim 37 , further comprising:
receiving, from the headset, additional image data for multiple poses of the headset;
identifying at least one second key point in the three-dimensional model of the environment at least partly represented in the additional image data;
performing bundle adjustment for each of the multiple poses by jointly optimizing (i) a reprojection error for the at least one key point based on the additional image data and (ii) an epipolar error for the at least one key point based on the additional image data; and
using results of the bundle adjustment for each of the multiple poses to perform at least one of: (i) updating the three-dimensional model, (ii) determining another position of the headset at each of the multiple poses, or (iii) determining other extrinsic parameters of the first camera and the second camera at each of the multiple poses.