Systems and methods for semi-supervised training using reprojected distance loss

1. A depth system for training a depth model for monocular depth estimation, comprising:

one or more processors;

a memory communicably coupled to the one or more processors and storing:

a network module including instructions that when executed by the one or more processors cause the one or more processors to:

generate, as part of training the depth model according to a supervised training stage, a depth map from a first image of a pair of training images using the depth model, wherein the pair of training images are separate frames depicting a scene from a monocular video, and wherein at least the first image includes corresponding depth data,

generate a transformation from the first image and a second image of the pair using a pose model, the transformation defining a relationship between the pair of training images; and

a training module including instructions that when executed by the one or more processors cause the one or more processors to compute a supervised loss based, at least in part, on reprojecting predicted pixels of the depth map and ground-truth depth of the depth data into a 3D space that is a reprojected area associated with the second image according to a project function using the transformation, wherein computing the supervised loss includes comparing the predicted pixels and the ground-truth depth within the reprojected area by reconstructing 3D points of the scene corresponding to a contextual view of the second image, and

update the depth model and the pose model together according to at least the supervised loss.

2. The depth system of claim 1 , wherein the training module includes instructions to compute the supervised loss including instructions to compute the supervised loss in combination with a self-supervised loss for the supervised training stage that is a second stage of training in a semi-supervised training process, and

wherein a first stage of training is self-supervised and occurs prior to the second stage.

3. The depth system of claim 2 , wherein the first stage is a self-supervised structure from motion (SfM) training process that accounts for motion of a camera between the training images to cause the depth model to learn how to infer depths without annotated training data, and

wherein the training module includes instructions to compute the supervised loss during the second stage including instructions to compute the supervised loss to refine the depth model using the depth data as selective dispersed ground truths providing limited supervision over depth estimates of the depth model.

4. The depth system of claim 2 , wherein the training module includes instructions to, during the first stage, produce first stage loss values from a first stage loss function that includes a photometric loss function and a depth smoothness loss function that separately account for pixel-level similarities and irregularities along edge regions between a synthesized image derived from depth predictions of the depth model and a target image of a respective one of the pairs.

5. The depth system of claim 1 ,

wherein the predicted pixels correspond with points in the scene as identified in the depth map and observed from the contextual view,

wherein the ground truth depth correspond with the depth data for the first image, and

wherein the training module includes instructions to compute the supervised loss including instructions to use a reprojected distance loss function to compare corresponding ones of the predicted pixels with the ground truth depth to generate a reprojected distance loss.

6. The depth system of claim 1 , wherein the depth data includes sparse LiDAR data comprising depth information from four beams that correspond with sparse locations in the training images, and

wherein training module includes instructions to perform a second stage to refine learned weights of the depth model and the pose model using the depth data to train the depth model on scale by accounting for scale aware differences between the depth maps and the sparse LiDAR data to improve scale awareness of the depth model in producing depth estimates.

7. The depth system of claim 1 , wherein the network module includes instructions to provide the depth model to infer distances from monocular images in a device,

wherein the training module includes instructions to generate a photometric loss by generating a synthesized version of the first image using the depth map and the transformation, and calculate the photometric loss according to a comparison of the synthesized version with the first image.

8. The depth system of claim 1 , wherein the depth model is a machine learning algorithm comprised of an encoder and a decoder that function together to generate depth estimates of a scene from a monocular image, and

wherein the pose model is a machine learning algorithm that performs a dimensional reduction of the training images to derive the transformation describing a change in pose between images within respective ones of the pairs.

9. A non-transitory computer-readable medium for training a depth model for monocular depth estimation and including instructions that when executed by one or more processors cause the one or more processors to:

generate a depth map from a first image of a pair of training images using the depth model, wherein the pair of training images are separate frames depicting a scene from a monocular video, and wherein at least the first image includes corresponding depth data;

generate a transformation from the first image and a second image of the pair using a pose model, the transformation defining a relationship between the pair of training images;

compute a supervised loss based, at least in part, on reprojecting predicted pixels of the depth map and ground-truth depth of the depth data into a 3D space that is a reprojected area associated with the second image according to a project function using the transformation, wherein computing the supervised loss includes comparing the predicted pixels and the ground-truth depth within the reprojected area by reconstructing 3D points of the scene corresponding to a contextual view of the second image; and

update the depth model and the pose model together according to at least the supervised loss.

10. The non-transitory computer-readable medium of claim 9 , wherein the instructions to compute the supervised loss include instructions to compute the supervised loss in combination with a self-supervised loss for a supervised training stage that is a second stage of training in a semi-supervised training process, and

wherein a first stage of training is self-supervised and occurs prior to the second stage.

11. The non-transitory computer-readable medium of claim 9 , wherein the ground truth depth corresponds with the depth data for the first image.

12. The non-transitory computer-readable medium of claim 9 ,

wherein the instructions to compute the supervised loss include instructions to use a reprojected distance loss function to compare corresponding ones of the predicted pixels with the ground truth depth to generate a reprojected distance loss.

13. A method of training a depth model for monocular depth estimation, comprising:

generating, as part of training the depth model according to a supervised training stage, a depth map from a first image of a pair of training images using the depth model, wherein the pair of training images are separate frames depicting a scene from a monocular video, and wherein at least the first image includes corresponding depth data;

generating a transformation from the first image and a second image of the pair using a pose model, the transformation defining a relationship between the pair of training images;

computing a supervised loss based, at least in part, on reprojecting predicted pixels of the depth map and ground-truth depth of the depth data into a 3D space that is a reprojected area associated with the second image according to a project function using the transformation, wherein computing the supervised loss includes comparing the predicted pixels and the ground-truth depth within the reprojected area by reconstructing 3D points of the scene corresponding to a contextual view of the second image; and

updating the depth model and the pose model together according to at least the supervised loss.

14. The method of claim 13 , wherein computing the supervised loss includes computing the supervised loss in combination with a self-supervised loss as the supervised training stage that is a second stage of training in a semi-supervised training process, and wherein a first stage of training is self-supervised and occurs prior to the second stage.

15. The method of claim 14 , wherein the first stage is a self-supervised structure from motion (SfM) training process that accounts for motion of a camera between the training images to cause the depth model to learn how to infer depths without annotated training data, and

wherein computing the supervised loss during the second stage includes computing the supervised loss to refine the depth model using the depth data as selective dispersed ground truths providing limited supervision over depth estimates of the depth model.

16. The method of claim 14 , wherein the first stage includes producing first stage loss values from a first stage loss function that includes a photometric loss function and a depth smoothness loss function that separately account for pixel-level similarities and irregularities along edge regions between a synthesized image derived from depth predictions of the depth model and a target image of a respective one of the pairs.

17. The method of claim 13 ,

wherein the predicted pixels correspond with points in the scene as identified in the depth map and observed from the contextual view,

wherein the ground truth depth correspond with the depth data for the first image, and

wherein computing the supervised loss includes using a reprojected distance loss function to compare corresponding ones of the predicted pixels with the ground truth depth to generate a reprojected distance loss.

18. The method of claim 13 , wherein the depth data includes sparse LiDAR data comprising depth information from four beams that correspond with sparse locations in the training images, and

wherein the second stage refines learned weights of the depth model and the pose model using the depth data to train the depth model on scale by accounting for scale aware differences between the depth maps and the sparse LiDAR data to improve scale awareness of the depth model in producing depth estimates.

19. The method of claim 13 , wherein the depth model is a machine learning algorithm comprised of an encoder and a decoder that function together to generate depth estimates of a scene from a monocular image, and

wherein the pose model is a machine learning algorithm that performs a dimensional reduction of the training images to derive the transformation describing a change in pose between images within respective ones of the pairs.

20. The method of claim 13 , further comprising:

providing the depth model to infer distances from monocular images in a device,

wherein computing the supervised loss further includes generating a photometric loss by generating a synthesized version of the first image using the depth map and the transformation, and calculating the photometric loss according to a comparison of the synthesized version with the first image.