Depth-guided structure-from-motion techniques

1. A computer-implemented method, comprising:

obtaining a plurality of image frames from a digital content item that corresponds to a scene;

performing a correspondence search on the plurality of image frames to determine a set of 2-D keypoints that are matched across the plurality of image frames;

obtaining a pretrained depth estimator;

determining, using the pretrained depth estimator and for the plurality of image frames, a plurality of dense depth maps;

determining, based on the plurality of dense depth maps, a corresponding set of depth priors for the set of 2-D keypoints;

determining, for the plurality of image frames, a plurality of 3-D points comprising at least a first 3-D point, wherein the first 3-D point is determined based at least in part on:

a first 2-D keypoint of a first image frame;

a first depth prior of the first 2-D keypoint; and

a first intrinsic matrix associated with the first image frame;

determining, based at least in part on the plurality of 3-D points, an image frame pair of a plurality of images, the image frame pair comprising a second image frame and a third image frame, wherein the second image frame is different from the third image frame;

determining, for the image frame pair, an initial camera pose;

obtaining a fourth image frame of the plurality of image frames, wherein the fourth image frame is different from the second image frame and the third image frame;

determining, based at least in part on the fourth image frame, an updated camera pose based at least in part on a first objective function, wherein the first objective function that is used to minimize a first loss determined based at least in part on:

a reprojection error for a set of inlier keypoints; and

a depth consistency error for the set of inlier keypoints; and

adding, based at least in part on the fourth image frame, one or more 3-D points to a point cloud via triangulation that is refined based at least in part on a second objective function that is used to minimize a second loss determined based at least in part on:

a reprojection error for the one or more 3-D points; and

a depth consistency error for the one or more 3-D points.

2. The computer-implemented method of claim 1 , wherein the set of depth prior are extracted from the plurality of dense depth maps using bilinear interpolation.

3. The computer-implemented method of claim 1 , further comprising determining a relative pose for the second image frame and the third image frame based at least in part on a geometric relationship between respective 2-D keypoints of the second image frame and the third image frame and their corresponding 3-D points of the plurality of 3-D points.

4. The computer-implemented method of claim 1 , further comprising:

obtaining a fifth image frame of the plurality of image frames, wherein the fifth image frame is different from the second image frame, the third image frame, and the fourth image frame;

determining, based at least in part on the fifth image frame, a second updated camera pose based at least in part on the first objective function; and

adding, based at least in part on the fifth image frame, additional one or more 3-D points to the point cloud via the triangulation that is refined based at least in part on the second objective function.

5. The computer-implemented method of claim 1 , wherein a majority of the plurality of image frames have a parallax of 1.0 or less.

6. A system, comprising:

one or more processors; and

memory storing executable instructions that, as a result of execution by the one or more processors, cause the system to:

obtain a plurality of image frames from a digital content item that corresponds to a scene;

determine, based at least in part on a correspondence search, a set of 2-D keypoints for the plurality of image frames;

obtain a depth estimator;

determine, based at least in part on the depth estimator, a plurality of dense depth maps for the plurality of image frames;

determine, based at least in part on the set of 2-D keypoints and the plurality of dense depth maps, a corresponding set of depth priors;

perform an initialization based at least in part on the set of 2-D keypoints and the corresponding set of depth priors to determine an initial image frame pair, the initial image frame pair comprises a first image frame and second image frame of the plurality of image frames;

determine a camera pose based at least in part on the first image frame and the second image frame; and

perform a depth-constrained optimization to incrementally update the camera pose based at least in part on the set of 2-D keypoints, and the corresponding set of depth priors.

7. The system of claim 6 , wherein the instructions to perform the initialization based at least in part on the set of 2-D keypoints and the corresponding set of depth priors to determine an initial image pair include instructions that, as a result of execution by the one or more processors, cause the system to:

determine, for the plurality of image frames, a plurality of 3-D points comprising at least a first 3-D point, wherein the first 3-D point is determined based at least in part on:

a first 2-D keypoint of a first image frame;

a first depth prior of the first 2-D keypoint; and

a first intrinsic matrix associated with the first image frame.

8. The system of claim 6 , wherein the initial image frame pair is selected based at least in part on how many 2-D keypoint correspondences are in the second image frame and the first image frame.

9. The system of claim 6 , wherein the set of depth prior are extracted from the plurality of dense depth maps using bilinear interpolation.

10. The system of claim 6 , wherein the executable instructions include further instructions that, as a result of execution by the one or more processors, further cause the system to:

determine a camera pose based at least in part on a first objective function, wherein the first objective function that is used to minimize a first loss determined based at least in part on:

a reprojection error for a set of inlier keypoints; and

a depth consistency error for the set of inlier keypoints.

11. The system of claim 6 , wherein the executable instructions include further instructions that, as a result of execution by the one or more processors, further cause the system to:

add one or more 3-D points to a point cloud via triangulation that is refined based at least in part on a second objective function that is used to minimize a second loss determined based at least in part on:

a reprojection error for the one or more 3-D points; and

a depth consistency error for the one or more 3-D points.

12. The system of claim 6 , wherein the depth estimator is a pretrained depth estimator.

13. The system of claim 6 , wherein a majority of the plurality of image frames have parallax of 1.0 or less.

14. A non-transitory computer-readable storage medium storing executable instructions that, as a result of being executed by one or more processors of a computer system, cause the computer system to at least:

obtain a plurality of image frames from a digital content item that corresponds to a scene;

determine, based at least in part on a correspondence search, a set of 2-D keypoints for the plurality of image frames;

obtain a depth estimator;

determine, based at least in part on the depth estimator, a plurality of dense depth maps for the plurality of image frames;

determine, based at least in part on the set of 2-D keypoints and the plurality of dense depth maps, a corresponding set of depth priors;

perform an initialization based at least in part on the set of 2-D keypoints and the corresponding set of depth priors to determine an initial image frame pair, the initial image frame pair comprises a first image frame and second image frame of the plurality of image frames;

determine a camera pose based at least in part on the first image frame and the second image frame; and

perform a depth-constrained optimization to incrementally update the camera pose based at least in part on the set of 2-D keypoints, and the corresponding set of depth priors.

15. The non-transitory computer-readable storage medium of claim 14 , wherein the instructions to perform the initialization based at least in part on the set of 2-D keypoints and the corresponding set of depth priors to determine an initial image pair include instructions that, as a result of execution by the one or more processors, cause the computer system to:

determine, for the plurality of image frames, a plurality of 3-D points comprising at least a first 3-D point, wherein the first 3-D point is determined based at least in part on:

a first 2-D keypoint of a first image frame;

a first depth prior of the first 2-D keypoint; and

a first intrinsic matrix associated with the first image frame.

16. The non-transitory computer-readable storage medium of claim 14 , wherein the initial image frame pair is selected based at least in part on how many 2-D keypoints correspondences are in the second image frame and the first image frame.

17. The non-transitory computer-readable storage medium of claim 14 , wherein the set of depth prior are extracted from the plurality of dense depth maps using bilinear interpolation.

18. The non-transitory computer-readable storage medium of claim 14 , wherein the executable instructions include further instructions that, as a result of execution by the one or more processors, further cause the computer system to:

determine a camera pose based at least in part on a first objective function, wherein the first objective function that is used to minimize a first loss determined based at least in part on:

a reprojection error for a set of inlier keypoints; and

a depth consistency error for the set of inlier keypoints.

19. The non-transitory computer-readable storage medium of claim 14 , wherein the executable instructions include further instructions that, as a result of execution by the one or more processors, further cause the computer system to:

add one or more 3-D points to a point cloud via triangulation that is refined based at least in part on a second objective function that is used to minimize a second loss determined based at least in part on:

a reprojection error for the one or more 3-D points; and

a depth consistency error for the one or more 3-D points.

20. The non-transitory computer-readable storage medium of claim 14 , wherein the depth estimator is a pretrained depth estimator.