Self-supervised multi-frame monocular depth estimation model
A multi-frame depth estimation model is disclosed. The model is trained and configured to receive an input image and an additional image. The model outputs a depth map for the input image based on the input image and the additional image. The model may extract a feature map for the input image and an additional feature map for the additional image. For each of a plurality of depth planes, the model warps the feature map to the depth plane based on relative pose between the input image and the additional image, the depth plane, and camera intrinsics. The model builds a cost volume from the warped feature maps for the plurality of depth planes. A decoder of the model inputs the cost volume and the input image to output the depth map.
1. A computer-implemented method comprising:
receiving a time series of images of a scene including a primary image and an additional image from an earlier time than the primary image, wherein the time series of images are monocular images derived from monocular video;
inputting the time series of images into a depth estimation model;
receiving, as output from the depth estimation model, a depth map of the primary image, the depth map generated based on a cost volume concatenating differences between a primary feature map of the primary image and a plurality of warped feature maps of the additional image for each of a plurality of depth planes, wherein receiving the depth map as output from the depth estimation model comprises;
generating a primary feature map for the primary image and an additional feature map for the additional image;
generating a warped feature map comprising a plurality of warped feature map layers, each warped feature map layer generated by warping the additional feature map to a plurality of depth planes based on (1) a depth plane of the plurality to which the feature map is being warped, (2) a relative pose between the primary image and the additional image, and (3) intrinsics of a camera used to capture the primary image and the additional image;
for each warped feature map layer, calculating a difference between the warped feature map layer and the primary feature map; and
building the cost volume by concatenating the differences between layers of the warped feature map and the primary feature map;
wherein the output is based on the cost volume and the primary feature map;
generating virtual content using the depth map; and
displaying an image of scene augmented with the virtual content.
2. The computer-implemented method of claim 1 , wherein the relative pose is determined by a convolutional neural network separately trained to determine the relative pose between two images.
3. The computer-implemented method of claim 1 , wherein the difference is an absolute difference.
4. The computer-implemented method of claim 1 , wherein the depth estimation model is trained by:
tuning a minimum depth plane and a maximum depth plane in the plurality of depth planes.
5. The computer-implemented method of claim 1 , wherein the depth estimation model is trained by:
inputting a training secondary image into a secondary depth estimation model to predict an estimated depth of the training secondary image;
generating a binary mask based on the estimated depth;
applying the binary mask to filter out unreliable pixels from the training secondary image; and
training the depth estimation model based on the training secondary image with filtered out unreliable pixels and a training input image.
6. The computer-implemented method of claim 1 , wherein the depth estimation model is trained by:
randomly, according to a set probability, replacing a training secondary image with a color augmented version of a training input image; and
training the depth estimation model based on the color augmented version of the training input image compared to the training input image.
7. The computer-implemented method of claim 1 , wherein the depth estimation model is trained by:
randomly, according to a set probability, setting a cost volume for a training secondary image to a constant value to generate a blank cost volume; and
training the depth estimation model based on the blank cost volume and a training input image.
8. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising:
receiving a time series of images of a scene including a primary image and an additional image from an earlier time than the primary image;
inputting the time series of images into a depth estimation model, wherein the time series of images are monocular images derived from monocular video;
receiving, as output from the depth estimation model, a depth map of the primary image, the depth map generated based on a cost volume concatenating differences between a primary feature map of the primary image and a plurality of warped feature maps of the additional image for each of a plurality of depth planes, wherein receiving the depth map as output from the depth estimation model comprises;
generating a primary feature map for the primary image and an additional feature map for the additional image;
generating a warped feature map comprising a plurality of warped feature map layers, each warped feature map layer generated by warping the additional feature map to a plurality of depth planes based on (1) a depth plane of the plurality to which the feature map is being warped, (2) a relative pose between the primary image and the additional image, and (3) intrinsics of a camera used to capture the primary image and the additional image;
for each warped feature map layer, calculating a difference between the warped feature map layer and the primary feature map; and
building the cost volume by concatenating the differences between layers of the warped feature map and the primary feature map;
wherein the output is based on the cost volume and the primary feature map;
generating virtual content using the depth map; and
displaying an image of scene augmented with the virtual content.
9. The non-transitory computer-readable storage medium of claim 8 , wherein the relative pose is determined by a convolutional neural network separately trained to determine the relative pose between two images.
10. The non-transitory computer-readable storage medium of claim 8 , wherein the difference is an absolute difference.
11. The non-transitory computer-readable storage medium of claim 8 , wherein the depth estimation model is trained by:
tuning a minimum depth plane and a maximum depth plane in the plurality of depth planes.
12. The non-transitory computer-readable storage medium of claim 8 , wherein the depth estimation model is trained by:
inputting a training secondary image into a secondary depth estimation model to predict an estimated depth of the training secondary image;
generating a binary mask based on the estimated depth;
applying the binary mask to filter out unreliable pixels from the training secondary image; and
training the depth estimation model based on the training secondary image with filtered out unreliable pixels and a training input image.
13. The non-transitory computer-readable storage medium of claim 8 , wherein the depth estimation model is trained by:
randomly, according to a set probability, replacing a training secondary image with a color augmented version of a training input image; and
training the depth estimation model based on the color augmented version of the training input image compared to the training input image.
14. The non-transitory computer-readable storage medium of claim 8 , wherein the depth estimation model is trained by:
randomly, according to a set probability, setting a cost volume for a training secondary image to a constant value to generate a blank cost volume; and
training the depth estimation model based on the blank cost volume and a training input image.