Self-supervised training of a depth estimation system
A method for training a depth estimation model and methods for use thereof are described. Images are acquired and input into a depth model to extract a depth map for each of the plurality of images based on parameters of the depth model. The method includes inputting the images into a pose decoder to extract a pose for each image. The method includes generating a plurality of synthetic frames based on the depth map and the pose for each image. The method includes calculating a loss value with an input scale occlusion and motion aware loss function based on a comparison of the synthetic frames and the images. The method includes adjusting the plurality of parameters of the depth model based on the loss value. The trained model can receive an image of a scene and generate a depth map of the scene according to the image.
1 . A non-transitory computer-readable storage medium storing:
a depth-pose hybrid model including a depth encoder and a pose decoder, wherein the pose decoder is configured to receive input based on output from the depth encoder, the depth-pose hybrid model manufactured by a process comprising:
acquiring a set of images;
applying the depth-pose hybrid model to generate depth maps and poses for the set of images based on parameters of the depth-pose hybrid model;
generating synthetic frames based on the depth maps and the poses for the set of images;
calculating a loss value with a loss function based on a comparison of the synthetic frames and the set of images; and
adjusting the parameters of the depth-pose hybrid model based on the loss value; and
instructions that, when executed by a computing device, cause the computing device to apply the depth-pose hybrid model to an image of a scene to generate a depth map of the scene.
2 . The non-transitory computer-readable storage medium of claim 1 , wherein the depth encoder and the pose decoder share at least some of the parameters of the depth-pose hybrid model.
3 . The non-transitory computer-readable storage medium of claim 1 , wherein generating the depth maps and the poses for the set of images comprises:
applying the depth encoder to the set of images to generate the depth maps; and
applying the pose decoder to the depth maps to generate the poses.
4 . The non-transitory computer-readable storage medium of claim 1 , wherein the set of images comprises monocular video with each image of the monocular video captured at a corresponding timestamp, wherein generating the synthetic frames comprises:
for images of the monocular video, generating synthetic frames at adjacent timestamps.
5 . The non-transitory computer-readable storage medium of claim 4 , wherein calculating the loss value with the loss function comprises:
calculating, for images of the monocular video, differentials between generated synthetic frames and images of the monocular video with matching timestamps.
6 . The non-transitory computer-readable storage medium of claim 1 , wherein:
the set of images includes a first image at a first timestamp, a second image at a second timestamp, and a third image at a third timestamp,
a first synthetic frame is generated at the second timestamp with the first image and a second synthetic frame is generated at the second timestamp with the third image, and
calculating the loss value with the loss function further comprises:
calculating a first set of differentials between the first synthetic frame and the second image and a second set of differentials between the second synthetic frame and the second image; and
for each pixel of the second image, identifying a minimum differential between the first set of differentials and the second set of differentials, wherein the loss value is based on the minimum differentials.
7 . The non-transitory computer-readable storage medium of claim 5 , wherein calculating the loss value with the loss function further comprises:
identifying one or more static features in the monocular video,
wherein the loss value is based on filtering out the differentials of the one or more static features.
8 . The non-transitory computer-readable storage medium of claim 7 , wherein identifying the one or more static features in the monocular video comprises:
identifying a region in a first image at a first timestamp of the monocular video and in a second image at a second timestamp of the monocular video;
determining whether an object is in a similar position between the first image and the second image; and
in response to determining that the object is in the similar position between the first image and the second image, defining the region as a static feature.
9 . The non-transitory computer-readable storage medium of claim 1 , wherein the set of images comprises stereoscopic image pairs with each stereoscopic image pair captured by a stereoscopic pair of cameras, wherein generating the synthetic frames comprises:
for each stereoscopic image pair, generating a synthetic frame based on a first image of the stereoscopic image pair.
10 . The non-transitory computer-readable storage medium of claim 9 , wherein adjusting the parameters of the depth-pose hybrid model based on the comparison of the synthetic frames and the set of images comprises:
calculating, for each stereoscopic image pair, a differential between the generated synthetic frame and a second image of the stereoscopic image pair; and
adjusting the parameters to minimize the differentials.
11 . A computer-implemented method comprising:
receiving an image of a scene;
inputting the image into a depth-pose hybrid model including a depth encoder and a pose decoder, wherein the pose decoder is configured to receive input based on output from the depth encoder, the depth-pose hybrid model trained with a process including:
acquiring a set of images;
inputting the set of images into the depth-pose hybrid model to extract depth maps and poses for the set of images based on parameters of the depth-pose hybrid model;
generating synthetic frames based on the depth maps and the poses for the set of images;
calculating a loss value with a loss function based on a comparison of the synthetic frames and the set of images; and
adjusting the parameters of the depth-pose hybrid model based on the comparison of the synthetic frames and the set of images; and
generating, by the depth-pose hybrid model, a depth map of the scene corresponding to the image of the scene.
12 . The computer-implemented method of claim 11 , wherein the depth encoder and the pose decoder share at least some of the parameters of the depth-pose hybrid model.
13 . The computer-implemented method of claim 11 , wherein generating the depth maps and the poses for the set of images comprises:
applying the depth encoder to the set of images to generate the depth maps; and
applying the pose decoder to the depth maps to generate the poses.
14 . The computer-implemented method of claim 11 , wherein the set of images comprises monocular video with each image of the monocular video captured at a corresponding timestamp, wherein generating the synthetic frames comprises:
for images of the monocular video, generating synthetic frames at adjacent timestamps.
15 . The computer-implemented method of claim 14 , wherein calculating the loss value with the loss function comprises:
calculating, for images of the monocular video, differentials between generated synthetic frames and images of the monocular video with matching timestamps.
16 . The computer-implemented method of claim 11 , wherein:
the set of images includes a first image at a first timestamp, a second image at a second timestamp, and a third image at a third timestamp,
a first synthetic frame is generated at the second timestamp with the first image and a second synthetic frame is generated at the second timestamp with the third image, and
calculating the loss value with the loss function further comprises:
calculating a first set of differentials between the first synthetic frame and the second image and a second set of differentials between the second synthetic frame and the second image; and
for each pixel of the second image, identifying a minimum differential between the first set of differentials and the second set of differentials, wherein the loss value is based on the minimum differentials.
17 . The computer-implemented method of claim 11 , further comprising:
displaying the image of the scene augmented with virtual content based on the depth map of the scene.
18 . A non-transitory computer-readable storage medium storing instructions that, when executed by a computing device, cause the computing device to perform operations comprising:
receiving an image of a scene;
inputting the image into a depth-pose hybrid model including a depth encoder and a pose decoder, wherein the pose decoder is configured to receive input based on output from the depth encoder, the depth-pose hybrid model trained with a process including:
acquiring a set of images;
inputting the set of images into the depth-pose hybrid model to extract depth maps and poses for the set of images based on parameters of the depth-pose hybrid model;
generating synthetic frames based on the depth maps and the poses for the set of images;
calculating a loss value with a loss function based on a comparison of the synthetic frames and the set of images; and
adjusting the parameters of the depth-pose hybrid model based on the comparison of the synthetic frames and the set of images; and
generating, by the depth-pose hybrid model, a depth map of the scene corresponding to the image of the scene.
19 . The non-transitory computer-readable storage medium of claim 18 , wherein:
the set of images includes a first image at a first timestamp, a second image at a second timestamp, and a third image at a third timestamp,
a first synthetic frame is generated at the second timestamp with the first image and a second synthetic frame is generated at the second timestamp with the third image, and
calculating the loss value with the loss function further comprises:
calculating a first set of differentials between the first synthetic frame and the second image and a second set of differentials between the second synthetic frame and the second image; and
for each pixel of the second image, identifying a minimum differential between the first set of differentials and the second set of differentials, wherein the loss value is based on the minimum differentials.
20 . The non-transitory computer-readable storage medium of claim 18 :
wherein the set of images comprises monocular video with each image of the monocular video captured at a corresponding timestamp;
wherein generating the synthetic frames comprises: for images of the monocular video, generating synthetic frames at adjacent timestamps;
wherein calculating the loss value with the loss function comprises: calculating, for images of the monocular video, differentials between generated synthetic frames and images of the monocular video with matching timestamps;
wherein calculating the loss value with the loss function further comprises: identifying one or more static features in the monocular video; and
wherein the loss value is based on filtering out the differentials of the one or more static features.
21 . The non-transitory computer-readable storage medium of claim 1 :
wherein the depth maps generated by the depth-pose hybrid model have lower resolutions than corresponding images of the set of images; and
wherein the process further comprises, prior to generating the synthetic frames, upsampling the depth maps to resolutions of corresponding images of the set of images.