Self-supervised training of a depth estimation system

1. A non-transitory computer-readable storage medium storing:

a trained depth model manufactured by a process comprising:

acquiring a set of images;

applying the depth model to generate depth maps for the images based on parameters of the depth model;

applying the images to a pose decoder to generate poses for the images;

generating synthetic frames based on the depth map and the poses for the images;

calculating a loss value with an input-scale occlusion-aware and motion-aware loss function based on a comparison of the synthetic frames and the images; and

adjusting the parameters of the depth model based on the loss value; and

instructions that, when executed by a computing device, cause the computing device to apply the trained depth model to an image of a scene to generate a depth map of the scene.

2. The non-transitory computer-readable storage medium of claim 1 , wherein the set of images comprises monocular video with each of image of the monocular video captured at a corresponding timestamp, wherein generating the synthetic frames comprises:

for images of the monocular video, generating synthetic frames at adjacent timestamps.

3. The non-transitory computer-readable storage medium of claim 2 , wherein calculating the loss value with the input-scale occlusion-aware and motion-aware loss function comprises:

calculating, for images of the monocular video, differentials between generated synthetic frames and images with matching timestamps.

4. The non-transitory computer-readable storage medium of claim 3 , wherein:

the set of images includes a first image at a first timestamp, a second image at a second timestamp, and a third image at a third timestamp,

a first synthetic frame is generated at the second timestamp with the first image and a second synthetic frame is generated at the second timestamp with the third image, and

calculating the loss value with the input-scale occlusion-aware and motion-aware loss function further comprises:

calculating a first set of differentials between the first synthetic frame and the second image and a second set of differentials between the second synthetic frame and the second image; and

for each pixel of the second image, identifying a minimum differential between the first set of differentials and the second set of differentials, wherein the loss value is based on the minimum differentials.

5. The non-transitory computer-readable storage medium of claim 3 , wherein calculating the loss value with the input-scale occlusion-aware and motion-aware loss function further comprises:

identifying one or more static features in the monocular video,

wherein the loss value is based on filtering out the differentials of the one or more static features.

6. The non-transitory computer-readable storage medium of claim 5 , wherein identifying one or more static features in the monocular video comprises:

identifying a region in a first image at a first timestamp of the monocular video and in a second image at a second timestamp of the monocular video;

determining whether the object is in a similar position between the first image and the second image; and

in response to determining that the object is in a similar position between the first image and the second image, defining the region as a static feature.

7. The non-transitory computer-readable storage medium of claim 1 , wherein the set of images are at a first resolution and the generated depth maps are at a second resolution that is lower than the first resolution, and wherein the process of manufacturing the depth model further comprises:

upsampling the generated depth maps into the first resolution from the second resolution.

8. The non-transitory computer-readable storage medium of claim 1 , wherein the set of images comprises stereoscopic image pairs with each stereoscopic image pair captured by a stereoscopic pair of cameras, wherein generating the synthetic frames comprises:

for each stereoscopic image pair, generating a synthetic frame based on a first image of the stereoscopic image pair.

9. The non-transitory computer-readable storage medium of claim 8 , wherein adjusting the parameters of the model based on the comparison of the synthetic frames and the set of images comprises:

calculating, for each stereoscopic image pair, a differential between the generated synthetic frame and a second image of the stereoscopic image pair; and

adjusting the parameters to minimize the differentials.

10. A computer-implemented method comprising:

receiving an image of a scene;

inputting the image into a trained depth model, the depth model trained with a process including:

acquiring a set of images;

inputting the images into the depth model to extract depth maps for the images based on parameters of the depth model;

inputting the images into a pose decoder to extract poses for the images;

generating synthetic frames based on the depth maps and the poses for the images;

calculating a loss value with an input-scale occlusion-aware and motion-aware loss function based on a comparison of the synthetic frames and the images; and

adjusting the parameters of the depth model based on the comparison of the synthetic frames and the images; and

generating, by the trained model, a depth map of the scene corresponding to the image of the scene.

11. The method of claim 10 , wherein the set of images comprises monocular video with each of image of the monocular video captured at a corresponding timestamp, wherein generating the synthetic frames comprises:

for images of the monocular video, generating synthetic frames at adjacent timestamps.

12. The method of claim 11 , wherein calculating the loss value with the input-scale occlusion-aware and motion-aware loss function comprises:

calculating, for images of the monocular video, differentials between generated synthetic frames and images with matching timestamps.

13. The method of claim 12 , wherein:

the set of images includes a first image at a first timestamp, a second image at a second timestamp, and a third image at a third timestamp,

a first synthetic frame is generated at the second timestamp with the first image and a second synthetic frame is generated at the second timestamp with the third image, and

calculating the loss value with the input-scale occlusion-aware and motion-aware loss function further comprises:

calculating a first set of differentials between the first synthetic frame and the second image and a second set of differentials between the second synthetic frame and the second image; and

for each pixel of the second image, identifying a minimum differential between the first set of differentials and the second set of differentials, wherein the loss value is based on the minimum differentials.

14. The method of claim 11 , wherein calculating the loss value with the input-scale occlusion-aware and motion-aware loss function further comprises:

identifying one or more static features in the monocular video with, wherein the loss value is based on filtering out the differentials of the one or more static features.

15. The method of claim 14 , wherein identifying one or more static features in the monocular video comprises:

identifying a region in a first image at a first timestamp of the monocular video and in a second image at a second timestamp of the monocular video;

determining whether the object is in a similar position between the first image and the second image; and

in response to determining that the object is in a similar position between the first image and the second image, defining the region as a static feature.

16. The method of claim 10 , wherein the set of images are at a first resolution and the extracted depth maps are at a second resolution that is lower than the first resolution, the process for training the depth model further comprising:

upsampling the extracted depth maps into the first resolution from the second resolution.

17. The method of claim 10 , wherein the set of images comprises stereoscopic image pairs with each stereoscopic image pair captured by a stereoscopic pair of cameras, wherein generating the synthetic frames comprises:

for each stereoscopic image pair, generating a synthetic frame based on a first image of the stereoscopic image pair.

18. The method of claim 17 , wherein adjusting the parameters of the model based on the comparison of the synthetic frames and the set of images comprises:

calculating, for each stereoscopic image pair, a differential between the generated synthetic frame and a second image of the stereoscopic image pair; and

adjusting the parameters to minimize the differentials.

19. The method of claim 10 , further comprising:

displaying the image of the scene augmented with virtual content based on the depth map of the scene.

20. A non-transitory computer-readable storage medium storing instructions that, when executed by a computing device, cause the computing device to perform operations comprising:

receiving an image of a scene;

inputting the image into a trained depth model, the depth model trained with a process including:

acquiring a set of images;

inputting the images into a depth encoder to extract abstract depth features for the images based on parameters of the depth encoder;

concatenating the abstract depth features for the images;

inputting the concatenated abstract depth features into a pose decoder to extract poses for the images;

generating synthetic frames based on parameters of the model and the poses for the images; and

adjusting the parameters of the model based on a comparison of the synthetic frames and the images; and

generating, by the trained model, a depth map of the scene corresponding to the image of the scene.