Self-supervised training of a depth estimation model using depth hints
A method for training a depth estimation model with depth hints is disclosed. For each image pair: for a first image, a depth prediction is determined by the depth estimation model and a depth hint is obtained; the second image is projected onto the first image once to generate a synthetic frame based on the depth prediction and again to generate a hinted synthetic frame based on the depth hint; a primary loss is calculated with the synthetic frame; a hinted loss is calculated with the hinted synthetic frame; and an overall loss is calculated for the image pair based on a per-pixel determination of whether the primary loss or the hinted loss is smaller, wherein if the hinted loss is smaller than the primary loss, then the overall loss includes the primary loss and a supervised depth loss between depth prediction and depth hint. The depth estimation model is trained by minimizing the overall losses for the image pairs.
1. A computer-implemented method for training a depth estimation model comprising:
for each image pair of a plurality of image pairs:
generating, by the depth estimation model, depth prediction values for pixels of a first image,
obtaining depth hint values for the pixels of the first image,
projecting a second image onto the first image based on the depth prediction values to generate a model synthetic frame,
projecting the second image onto the first image based on the depth hint values to generate a hinted synthetic frame, and
calculating an overall loss for the image pair as a combination of a primary loss, based on a comparison of the model synthetic frame to the first image, and a hinted loss, based on a comparison of the hinted synthetic frame to the first image; and
adjusting parameters of the depth estimation model to minimize the overall losses of the image pairs.
2. The computer-implemented method of claim 1 , wherein each image pair is a true stereoscopic image pair captured by a pair of cameras.
3. The computer-implemented method of claim 1 , wherein each image pair is a pseudo stereoscopic image pair, wherein the first image and the second image are temporally distinct frames captured by a single camera.
4. The computer-implemented method of claim 1 , wherein, for each image pair, projecting the second image onto the first image is further based on a pose between the first image and the second image.
5. The computer-implemented method of claim 1 , wherein, for each image pair, projecting the second image onto the first image is further based on a first set of camera intrinsic parameters for the first image and a second set of camera intrinsic parameters for the second image.
6. The computer-implemented method of claim 1 , wherein the primary loss value at a pixel is based on a differential between a pixel of the model synthetic frame and a pixel of the first image, and wherein the hinted loss value at the pixel is based on a differential between a pixel of the hinted synthetic frame and the pixel of the first image.
7. The computer-implemented method of claim 6 ,
wherein contribution of a given pixel to the overall loss is the primary loss value of the given pixel, if the primary loss value is smaller than or equal to the hinted loss value at the given pixel; and
wherein contribution of the given pixel to the overall loss is a sum of the primary loss value of the given pixel and a differential between a depth prediction value and a depth hint value at the given pixel, if the hinted loss value is smaller than the primary loss value at the given pixel.
8. The computer-implemented method of claim 1 , wherein the depth hint values are generated by a stereo depth estimation model configured to input the image pair and to output the depth hint values based on the image pair.
9. The computer-implemented method of claim 1 , wherein the depth hint values are generated by a simultaneous-localization-and-mapping (SLAM) module based on the image pair.
10. A computer-implemented method for predicting a depth map from an image of a scene using a depth estimation model, the method comprising:
receiving the image of the scene;
inputting the image into the depth estimation model that was trained by a process comprising:
for each image pair of a plurality of image pairs:
generating, by the depth estimation model, depth prediction values for pixels of a first image,
obtaining depth hint values for the pixels of the first image,
projecting a second image onto the first image based on the depth prediction values to generate a model synthetic frame,
projecting the second image onto the first image based on the depth hint values to generate a hinted synthetic frame, and
calculating an overall loss for the image pair as a combination of a primary loss, based on a comparison of the model synthetic frame to the first image, and a hinted loss, based on a comparison of the hinted synthetic frame to the first image; and
adjusting parameters of the depth estimation model to minimize the overall losses of the image pairs; and
generating, by the depth estimation model, the depth map of the scene.
11. The computer-implemented method of claim 10 , wherein each image pair is a true stereoscopic image pair captured by a pair of cameras.
12. The computer-implemented method of claim 11 , wherein each image pair is a pseudo stereoscopic image pair, wherein the first image and the second image are temporally distinct frames captured by a single camera.
13. The computer-implemented method of claim 11 , wherein, for each image pair, projecting the second image onto the first image is further based on a pose between the first image and the second image.
14. The computer-implemented method of claim 11 , wherein, for each image pair, projecting the second image onto the first image is further based on a first set of camera intrinsic parameters for the first image and a second set of camera intrinsic parameters for the second image.
15. The computer-implemented method of claim 11 , wherein the primary loss value at a pixel is based on a differential between a pixel of the model synthetic frame and a pixel of the first image, and wherein the hinted loss value at the pixel is based on a differential between a pixel of the hinted synthetic frame and the pixel of the first image.
16. The computer-implemented method of claim 15 ,
wherein contribution of a given pixel to the overall loss is the primary loss value of the given pixel, if the primary loss value is smaller than or equal to the hinted loss value at the given pixel; and
wherein contribution of the given pixel to the overall loss is a sum of the primary loss value of the given pixel and a differential between a depth prediction value and a depth hint value at the given pixel, if the hinted loss value is smaller than the primary loss value at the given pixel.
17. The computer-implemented method of claim 11 , wherein the depth hint values are generated by a stereo depth estimation model configured to input the image pair and to output the depth hint values based on the image pair.
18. The computer-implemented method of claim 11 , wherein the depth hint values are generated by a simultaneous-localization-and-mapping (SLAM) module based on the image pair.
19. The computer-implemented method of claim 11 , further comprising:
generating a virtual object to present in the image of the scene based on the depth map of the scene; and
displaying, on an electronic display, the image of the scene augmented with the virtual object.
20. The computer-implemented method of claim 11 , further comprising:
determining navigation instructions for an autonomous vehicle based on the depth map; and
navigating the autonomous vehicle based on the navigation instructions.
21. A non-transitory computer-readable storage medium storing instructions for predicting a depth map from an image of a scene using a depth estimation model, the instructions that, when executed by a processor, cause the processor to perform operations comprising:
receiving the image of the scene;
inputting the image into the depth estimation model that was trained by a process comprising:
for each image pair of a plurality of image pairs:
generating, by the depth estimation model, depth prediction values for pixels of a first image,
obtaining depth hint values for the pixels of the first image,
projecting a second image onto the first image based on the depth prediction values to generate a model synthetic frame,
projecting the second image onto the first image based on the depth hint values to generate a hinted synthetic frame, and
calculating an overall loss for the image pair as a combination of a primary loss, based on a comparison of the model synthetic frame to the first image, and a hinted loss, based on a comparison of the hinted synthetic frame to the first image; and
adjusting parameters of the depth estimation model to minimize the overall losses of the image pairs; and
generating, by the depth estimation model, the depth map of the scene.