Pseudo RGB-D for self-improving monocular slam and depth prediction

1. A computer-implemented method executed on a processor for improving geometry-based monocular structure from motion (SfM) by exploiting depth maps predicted by convolutional neural networks (CNNs), the method comprising:

capturing a sequence of RGB images from an unlabeled monocular video stream obtained by a monocular camera;

feeding the RGB images into a CNN-based depth estimation/refinement module network;

outputting depth maps;

feeding the depth maps and the RGB images to a pose estimation/refinement network, the depths maps and the RGB images collectively defining pseudo RGB-D images;

outputting camera poses and point clouds; and

constructing a 3D map of a surrounding environment displayed on a visualization device,

wherein appearance-based losses are computed on narrow baselines and purely geometric losses are computed on wide baselines such that a total loss for fine-tuning the depth estimation/refinement network is a weighted sum of narrow baseline losses, wide baseline losses, and depth consistency losses given as:

=α +β c +γ c +μ( + )

where α, β, γ, and μ are loss terms, is a photometric loss, c is a smoothness loss, c is a depth consistency loss and , , and are symmetric transfer losses between frames.

2. The method of claim 1 , wherein common tracked keypoints from neighboring keyframes are employed.

3. The method of claim 2 , wherein the symmetric transfer losses and the depth consistency loss are imposed on the common tracked keypoints.

4. The method of claim 3 , wherein the symmetric transfer losses are given as:

( w )=| d c→k1 i ( w )− d k1 i ( w )|+| d k1→c i ( w )− d c i ( w )|

where d k1 i ( w ) and d c i ( w ) are the depth values from the depth network, d c→k1 i ( w ) and d k1→c i ( w ) are the transferred depth values, k 1 and k 2 are two neighboring keyframes of a current frame c, and w represents the depth network parameters.

5. The method of claim 4 , wherein the depth consistency loss is given as:

D

c

=

Σ

i

∈

𝒳

⁢



d

c

i

⁡

(

w

)

-

d

c

i

⁡

(

S

⁢

L

⁢

AM

)





𝒳



where X represents a set of common tracked keypoints, c is a current frame, w is a depth network parameter, d c i ( w ) is a depth value from the depth network, and d c i (SLAM) is a depth value from SLAM.

6. The method of claim 5 , wherein the photometric loss between a synthesized frame and a current frame is given as:

=pe( I c+1→c ( d c ( w ), T c+1→c SLAM ,K ), I c )+pe( I c+1→c ( d c ( w ), T c−1→c SLAM ,K ), I c ),

where I c is a current keyframe, I c−i and I c+1 are adjacent frames, K is a camera intrinsic matrix, w is the depth network parameter, d c (w) is a network-predicted depth value, and T c−1→c SLAM and T c+1→c SLAM represent relative camera poses between two frames.

7. The method of claim 6 , wherein the unlabeled monocular video stream is used in both training and inference stages.

8. A non-transitory computer-readable storage medium comprising a computer-readable program for improving geometry-based monocular structure from motion (SfM) by exploiting depth maps predicted by convolutional neural networks (CNNs), wherein the computer-readable program when executed on a computer causes the computer to perform the steps of:

capturing a sequence of RGB images from an unlabeled monocular video stream obtained by a monocular camera;

feeding the RGB images into a CNN-based depth estimation/refinement network;

outputting depth maps;

feeding the depth maps and the RGB images to a pose estimation/refinement network, the depths maps and the RGB images collectively defining pseudo RGB-D images;

outputting camera poses and point clouds; and

constructing a 3D map of a surrounding environment displayed on a visualization device

wherein appearance-based losses are computed on narrow baselines and purely geometric losses are computed on wide baselines such that a total loss for fine-tuning the depth estimation/refinement network is a weighted sum of narrow baseline losses, wide baseline losses, and depth consistency losses given as:

=α +β c +γ c +μ( + )

where α,β, γ, and μ are loss terms, is a photometric loss, c is a smoothness loss, c is a depth consistency loss and , , and are symmetric transfer losses between frames.

9. The non-transitory computer-readable storage medium of claim 8 , wherein common tracked keypoints from neighboring keyframes are employed.

10. The non-transitory computer-readable storage medium of claim 9 , wherein the symmetric transfer losses and the depth consistency loss are imposed on the common tracked keypoints.

11. The non-transitory computer-readable storage medium of claim 10 , wherein the symmetric transfer is losses are given as:

( w )=| d c→k1 i ( w )− d k1 i ( w )|+| d k1→c i ( w )− d c i ( w )|

where d k1 i ( w ) and d c i ( w ) are the depth values from the depth network, d c→k1 i ( w ) and d k1→c i ( w ) are the transferred depth values, k 1 and k 2 are two neighboring keyframes of a current frame c, and w represents the depth network parameters.

12. The non-transitory computer-readable storage medium of claim 11 , wherein the depth consistency loss is given as:

D

c

=

Σ

i

∈

𝒳

⁢



d

c

i

⁡

(

w

)

-

d

c

i

⁡

(

S

⁢

L

⁢

AM

)





𝒳



where X represents a set of common tracked keypoints, c is a current frame, w is a depth network parameter, d c i ( w ) is a depth value from the depth network, and d c i (SLAM) is a depth value from SLAM.

13. The non-transitory computer-readable storage medium of claim 12 , wherein the photometric loss between a synthesized frame and a current frame is given as:

=pe( I c+1→c ( d c ( w ), T c+1→c SLAM ,K ), I c )+pe( I c+1→c ( d c ( w ), T c−1→c SLAM ,K ), I c ),

where I c is a current keyframe, I c−i and I c+1 are adjacent frames, K is a camera intrinsic matrix, w is the depth network parameter, d c (w) is a network-predicted depth value, and T c−1→c SLAM and T c+1→c SLAM represent relative camera poses between two frames.

14. The non-transitory computer-readable storage medium of claim 13 , wherein the unlabeled monocular video stream is used in both training and inference stages.

15. A system for improving geometry-based monocular structure from motion (SfM) by exploiting depth maps predicted by convolutional neural networks (CNNs), the system comprising:

a memory; and

one or more processors in communication with the memory configured to:

capture a sequence of RGB images from an unlabeled monocular video stream obtained by a monocular camera;

feed the RGB images into a CNN-based depth estimation/refinement network;

output depth maps;

feed the depth maps and the RGB images to a pose estimation/refinement network, the depths maps and the RGB images collectively defining pseudo RGB-D images;

output camera poses and point clouds; and

construct a 3D map of a surrounding environment displayed on a visualization device,

wherein appearance-based losses are computed on narrow baselines and purely geometric losses are computed on wide baselines such that a total loss for fine-tuning the depth estimation/refinement network is a weighted sum of narrow baseline losses, wide baseline losses, and depth consistency losses given as:

=α +β c +γ c +μ( + )

where α,β, γ, and μ are loss terms, is a photometric loss, c is a smoothness loss, c is a depth consistency loss and , , and are symmetric transfer losses between frames.

16. The system of claim 15 , wherein common tracked keypoints from neighboring keyframes are employed.

17. The system of claim 16 , wherein the symmetric transfer losses and the depth consistency loss are imposed on the common tracked keypoints.

18. The system of claim 17 , wherein the symmetric transfer losses are given as:

( w )=| d c→k1 i ( w )− d k1 i ( w )|+| d k1→c i ( w )− d c i ( w )|

where d k1 i ( w ) and d c i ( w ) are the depth values from the depth network, d c→k1 i ( w ) and d k1→c i ( w ) are the transferred depth values, k 1 and k 2 are two neighboring keyframes of a current frame c, and w represents the depth network parameters.

19. The system of claim 18 , wherein the depth consistency loss is given as:

D

c

=

Σ

i

∈

𝒳

⁢



d

c

i

⁡

(

w

)

-

d

c

i

⁡

(

S

⁢

L

⁢

AM

)





𝒳



where X represents a set of common tracked keypoints, c is a current frame, w is a depth network parameter, d c i ( w ) is a depth value from the depth network, and d c i (SLAM) is a depth value from SLAM.

20. The system of claim 19 , wherein the photometric loss between a synthesized frame and a current frame is given as:

=pe( I c+1→c ( d c ( w ), T c+1→c SLAM ,K ), I c )+pe( I c+1→c ( d c ( w ), T c−1→c SLAM ,K ), I c ),

where I c is a current keyframe, I c−i and I c+1 are adjacent frames, K is a camera intrinsic matrix, w is the depth network parameter, d c (w) is a network-predicted depth value, and T c−1→c SLAM and T c+1→c SLAM represent relative camera poses between two frames.