Self-supervised training of a depth estimation model using depth hints

1. A computer-implemented method comprising:

receiving an image of a scene;

inputting the image into a depth estimation model that was trained by a process comprising:

accessing training image data comprising a plurality of pseudo-stereoscopic image pairs, each pseudo-stereoscopic image pair comprising a first image and a second image;

for each image pair:

generating, by the depth estimation model, depth prediction values for pixels of the first image;

obtaining a camera pose hint for the first image;

projecting the second image onto the first image using the depth prediction values for the pixels of the first image to generate a model synthetic frame;

projecting the second image onto the first image using the camera pose hint for the first image to generate a hinted synthetic frame;

calculating primary loss values corresponding to the pixels of the first image based on a comparison of the pixels of the model synthetic frame and the pixels of the first image;

calculating hinted loss values corresponding to the pixels of the first image based on a comparison of the pixels of the hinted synthetic frame and the pixels of the first image; and

calculating an overall loss for the image pair, wherein, responsive to the hinted loss value corresponding to a given pixel of the first image being smaller than the primary loss value corresponding to the given pixel, a contribution of the given pixel to the overall loss is determined using the primary loss value corresponding to the given pixel and a supervised loss value, the supervised loss value determined using a depth prediction value for the given pixel and a depth value for the given pixel determined using the camera pose hint;

adjusting the parameters of the depth estimation model based on the overall losses of the pseudo-stereoscopic image pairs; and

generating, by the depth estimation model, a depth map of the scene corresponding to the image of the scene.

2. The method of claim 1 , wherein, responsive to the hinted loss value corresponding to a given pixel of the first image being greater than the primary loss value corresponding to the given pixel, the contribution of the given pixel to the overall loss does not include the supervised loss value.

3. The method of claim 1 wherein the process for training the depth estimation model further comprises:

obtaining depth hint values for the pixels of the first image;

projecting the second image onto the first image based on the depth hint values for the pixels of the first image to generate an additional hinted synthetic frame;

calculating additional hinted loss values for the depth hint values based on a comparison of the pixels of the additional hinted synthetic frame and the pixels of the first image; and

calculating the overall loss for the image pair using the additional hinted loss values.

4. The method of claim 1 , wherein the depth value for the given pixel is determined using a disparity map generated for the first image and the hinted synthetic frame using the camera pose hint.

5. The method of claim 1 , wherein the first image and the second image are temporally distinct frames captured by a single camera.

6. The method of claim 1 , wherein projecting the second image onto the first image based on the depth prediction values is further based on a pose between the first image and the second image.

7. The method of claim 1 , wherein the primary loss value at a pixel according to a depth prediction value is based on a differential between the pixel of the model synthetic frame and the pixel of the first image, and wherein the hinted loss value at the pixel according to a depth hint value is based on a differential between the pixel of the hinted synthetic frame and the pixel of the first image.

8. The method of claim 1 ,

wherein the contribution of a given pixel to the overall loss is the primary loss value of the given pixel if the primary loss value is smaller than or equal to the hinted loss value at the given pixel; and

wherein the contribution of the given pixel to the overall loss is a sum of the primary loss value of the given pixel and a differential between the depth prediction value and the depth value determined using the camera pose hint for the given pixel, if the hinted loss value is smaller than the primary loss value at the given pixel.

9. The method of claim 1 , wherein the camera pose hint is generated by a pose estimation model configured to input the image pair and to output the camera pose hint based on the image pair.

10. A computer-implemented method for training a depth estimation model comprising:

accessing training image data comprising a plurality of pseudo-stereoscopic image pairs, each pseudo-stereoscopic image pair comprising a first image and a second image;

for each image pair:

generating, by the depth estimation model, depth prediction values for pixels of the first image;

obtaining a camera pose hint for the first image;

projecting the second image onto the first image using the depth prediction values for the pixels of the first image to generate a model synthetic frame;

projecting the second image onto the first image using the camera pose hint for the first image to generate a hinted synthetic frame;

calculating primary loss values corresponding to the pixels of the first image based on a comparison of the pixels of the model synthetic frame and the pixels of the first image;

calculating hinted loss values corresponding to the pixels of the first image based on a comparison of the pixels of the hinted synthetic frame and the pixels of the first image; and

calculating an overall loss for the image pair, wherein, responsive to the hinted loss value corresponding to a given pixel of the first image being smaller than the primary loss value corresponding to the given pixel, a contribution of the given pixel to the overall loss is determined using the primary loss value corresponding to the given pixel and a supervised loss value, the supervised loss value determined using a depth prediction value for the given pixel and a depth value for the given pixel determined using the camera pose hint; and

adjusting the parameters of the depth estimation model based on the overall losses of the pseudo-stereoscopic image pairs.

11. The method of claim 10 , wherein, responsive to the hinted loss value corresponding to a given pixel of the first image being greater than the primary loss value corresponding to the given pixel, the contribution of the given pixel to the overall loss does not include the supervised loss value.

12. The method of claim 10 , wherein the process for training the depth estimation model further comprises:

obtaining depth hint values for the pixels of the first image;

projecting the second image onto the first image based on the depth hint values for the pixels of the first image to generate an additional hinted synthetic frame;

calculating additional hinted loss values for the depth hint values based on a comparison of the pixels of the additional hinted synthetic frame and the pixels of the first image; and

calculating the overall loss for the image pair using the additional hinted loss values.

13. The method of claim 10 , wherein the depth value for the given pixel is determined using a disparity map generated for the first image and the hinted synthetic frame using the camera pose hint.

14. The method of claim 10 , wherein the first image and the second image are temporally distinct frames captured by a single camera.

15. The method of claim 10 , wherein projecting the second image onto the first image based on the depth prediction values is further based on a pose between the first image and the second image.

16. The method of claim 10 , wherein the primary loss value at a pixel according to a depth prediction value is based on a differential between the pixel of the model synthetic frame and the pixel of the first image, and wherein the hinted loss value at the pixel according to a depth hint value is based on a differential between the pixel of the hinted synthetic frame and the pixel of the first image.

17. The method of claim 10 ,

wherein the contribution of a given pixel to the overall loss is the primary loss value of the given pixel if the primary loss value is smaller than or equal to the hinted loss value at the given pixel; and

wherein the contribution of the given pixel to the overall loss is a sum of the primary loss value of the given pixel and a differential between the depth prediction value and the depth value determined using the camera pose hint for the given pixel, if the hinted loss value is smaller than the primary loss value at the given pixel.

18. The method of claim 10 , wherein the camera pose hint is generated by a pose estimation model configured to input the image pair and to output the camera pose hint based on the image pair.

19. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising:

inputting the image into a depth estimation model that was trained by a process comprising:

accessing training image data comprising a plurality of pseudo-stereoscopic image pairs, each pseudo-stereoscopic image pair comprising a first image and a second image;

for each image pair:

generating, by the depth estimation model, depth prediction values for pixels of the first image;

obtaining a camera pose hint for the first image;

projecting the second image onto the first image using the depth prediction values for the pixels of the first image to generate a model synthetic frame;

projecting the second image onto the first image using the camera pose hint for the first image to generate a hinted synthetic frame;

calculating primary loss values corresponding to the pixels of the first image based on a comparison of the pixels of the model synthetic frame and the pixels of the first image;

calculating hinted loss values corresponding to the pixels of the first image based on a comparison of the pixels of the hinted synthetic frame and the pixels of the first image; and

calculating an overall loss for the image pair, wherein, responsive to the hinted loss value corresponding to a given pixel of the first image being smaller than the primary loss value corresponding to the given pixel, a contribution of the given pixel to the overall loss is determined using the primary loss value corresponding to the given pixel and a supervised loss value, the supervised loss value determined using a depth prediction value for the given pixel and a depth value for the given pixel determined using the camera pose hint;

adjusting the parameters of the depth estimation model based on the overall losses of the pseudo-stereoscopic image pairs; and

generating, by the depth estimation model, a depth map of the scene corresponding to the image of the scene.

20. The storage medium of claim 19 , wherein the process for training the depth estimation model further comprises:

obtaining depth hint values for the pixels of the first image;

projecting the second image onto the first image based on the depth hint values for the pixels of the first image to generate an additional hinted synthetic frame;

calculating additional hinted loss values for the depth hint values based on a comparison of the pixels of the additional hinted synthetic frame and the pixels of the first image; and

calculating the overall loss for the image pair using the additional hinted loss values.