Methods and apparatus for augmenting dense depth maps using sparse data
Systems, apparatus, and methods for augmenting dense depth maps using sparse data. Various embodiments combine single-image depth estimation (SIDE) techniques with structure-from-motion techniques for improved depth accuracy. In some examples, a machine learning (ML) model is used to generate a dense depth map based on one or more frames/images of a video. Structure-from-motion (SfM) analysis is performed on the video to determine depth information from camera movement in the video. The structure-from-motion techniques may generate more accurate data than the ML model, however, the data from the ML model may be denser compared with the SFM depth data. The dense ML model depth map may be augmented by the SFM depth data. Augmentation may include fitting the relative depths determined by the ML model depth map to the absolute depth in the SFM data resulting in more accurate dense depth information.
1 . A method of generating a depth map, comprising:
generating a first depth map of a frame of a video using a first technique;
generating a second depth map of the frame of the video using a second technique;
adjusting the second depth map based on the first depth map; and
performing local re-mapping of second depth values of the second depth map comprising:
determining depth map regions in the second depth map;
determining local adjustments for each of the depth map regions; and
applying the local adjustments to the second depth map.
2 . The method of claim 1 , further comprising:
generating a three-dimensional (3D) point cloud, where generating the first depth map is based on the 3D point cloud.
3 . The method of claim 2 , where generating the 3D point cloud comprises performing a structure-from-motion solve on the video.
4 . The method of claim 1 , where the first technique is a single-image depth estimation technique.
5 . The method of claim 1 , where:
the first depth map is a sparse depth map, and
the second depth map is a dense depth map, the sparse depth map having fewer depth values in the depth map than the dense depth map.
6 . The method of claim 1 , where adjusting the second depth map comprises shifting second depth values of the second depth map to minimize a difference of the second depth values of the second depth map and co-located first depth values of the first depth map.
7 . The method of claim 1 , where adjusting the second depth map based on the first depth map comprises re-mapping depth values in the second depth map to fit co-located depth values in the first depth map.
8 . The method of claim 1 , where adjusting the second depth map based on the first depth map comprises re-mapping the second depth values of the second depth map based on first depth values of the first depth map.
9 . The method of claim 1 , where:
adjusting the second depth map based on the first depth map produces an adjusted second depth map with adjusted second depth values, and
determining the local adjustments comprises determining a difference between the adjusted second depth values and co-located first depth values of the first depth map.
10 . The method of claim 1 , where determining the depth map regions in the second depth map is based on pixel locations in the first depth map.
11 . A post-processing device, comprising:
a processor; and
a non-transitory computer-readable medium comprising a set of instructions that, when executed by the processor, causes the processor to:
receive a video;
generate a sparse depth map for a frame of the video;
generate a dense depth map for the frame of the video;
adjust first depth values of the dense depth map based on second depth values of the sparse depth map generating an adjusted dense depth map;
determine differences between third depth values of the adjusted dense depth map and second depth values of the sparse depth map;
determining local adjustments corresponding to a plurality of regions of the adjusted dense depth map based on the differences; and
applying the local adjustments to the adjusted dense depth map.
12 . The post-processing device of claim 11 , where the set of instructions further causes the processor to generate a 3D point cloud based on a plurality of frames of the video, the 3D point cloud being used to generate the sparse depth map for the frame of the video.
13 . The post-processing device of claim 12 , where the set of instructions further causes the processor to refine the 3D point cloud by removing outlier values.
14 . The post-processing device of claim 12 , where the set of instructions further causes the processor to:
determine objects in the 3D point cloud; and
refine the 3D point cloud by removing moving objects from the 3D point cloud.
15 . The post-processing device of claim 11 , where the video comprises a plurality of frames and each of the plurality of frames is associated with capture metadata comprising at least one of capture location metadata and capture orientation metadata.
16 . The post-processing device of claim 15 , where the set of instructions further causes the processor to generate a 3D point cloud based on the plurality of frames of the video and the capture metadata, the 3D point cloud being used to generate the sparse depth map for the frame of the video.
17 . The post-processing device of claim 11 , where the set of instructions further causes the processor to perform a structure-from-motion analysis on at least a portion of the video, where generating the sparse depth map is based on the structure-from-motion analysis.
18 . A method of post-processing video, comprising:
receiving a video;
generating a three-dimensional (3D) point cloud corresponding to depths of objects in a plurality of frames of the video using a structure-from-motion technique;
generating a sparse depth map for a frame of the plurality of frames of the video based on the 3D point cloud;
generating a dense depth map for the frame of the video based on a single-image depth estimation technique;
adjusting the dense depth map based on minimizing depth differences between the dense depth map and the sparse depth map creating an adjusted dense depth map;
determining differences between first depth values of the adjusted dense depth map and second depth values of the sparse depth map;
determining local adjustments corresponding to a plurality of regions of the adjusted dense depth map based on the differences; and
applying the local adjustments to the adjusted dense depth map.
19 . The method of claim 18 , where:
the sparse depth map comprises first depth values corresponding to a first number of pixel locations of the frame of the video,
the dense depth map comprises second depth values corresponding to a second number of pixel locations of the frame of the video, and
the second number of pixel locations is greater than the first number of pixel locations.
20 . The method of claim 18 , further comprising determining the plurality of regions of the adjusted dense depth map based on pixel locations in the sparse depth map.