3D model reconstruction and scale estimation
Embodiments include systems and methods for creation of a 3D mesh from a video stream or a sequence of frames. A sparse point cloud is first created from the video stream, which is then densified per frame by comparison with spatially proximate frames. A 3D mesh is then created from the densified depth maps, and the mesh is textured by projecting the images from the video stream or sequence of frames onto the mesh. Metric scale of the depth maps may be estimated where direct measurements are not able to be measured or calculated using a machine learning depth estimation network.
1. A method, comprising:
receiving, at a computing device, a sequence of frames of a scene captured by a camera;
passing, by the computing device, each frame of the sequence of frames through a depth estimation network to obtain an estimated depth map;
estimating, by the computing device, a camera pose for the camera from the sequence of frames;
generating, by the computing device, a sparse depth map for each frame from the sequence of frames, each sparse depth map comprised of at least one 3D point;
rendering, by the computing device from the sparse depth map and estimated camera pose, a camera view depth map representing the camera view; and
fitting, by the computing device, the camera view depth map to the estimated depth map to obtain a sparse depth map with an estimated metric scale by:
resizing either or both of the camera view depth map and the estimated depth map;
comparing the resized camera view depth map and the estimated depth map to obtain a disparity measure;
repeating the resizing and comparing until the disparity measure is reduced to a minimum;
obtaining a scale factor that corresponds to the minimum disparity measure; and
applying the scale factor to the sparse depth map.
2. The method of claim 1 , wherein the estimated depth map comprises predicted per pixel depths.
3. The method of claim 2 , wherein passing each frame of the sequence of frames through a depth estimation network comprises passing each frame of the sequence of frames through a monocular depth estimation network.
4. The method of claim 1 , wherein the camera pose comprises camera intrinsics.
5. The method of claim 1 , wherein comparing the resized camera view depth map and the estimated depth map to obtain the scale factor comprises performing a Procrustes analysis.
6. A non-transitory computer readable medium (CRM) comprising instructions that, when executed by a processor of an apparatus, cause the apparatus to:
receive a sequence of frames of a scene captured by a camera;
pass each frame of the sequence of frames through a depth estimation network to obtain an estimated depth map;
estimate a camera pose for the camera from the sequence of frames;
generate a sparse depth map for each frame from the sequence of frames, each sparse depth map comprised of at least one 3D point;
render, from the sparse depth map and estimated camera pose, a camera view depth map representing the camera view; and
fit the camera view depth map to the estimated depth map to obtain a sparse depth map with an estimated metric scale, the instructions to perform the fit further comprising instructions to:
resize either or both of the camera view depth map and the estimated depth map;
compare the resized camera view depth map and the estimated depth map to obtain a disparity measure;
repeat the resizing and comparing until the disparity measure is reduced to a minimum;
obtain a scale factor that corresponds to the minimum disparity measure; and
apply the scale factor to the sparse depth map.
7. The CRM of claim 6 , wherein the estimated depth map comprises predicted per pixel depths.
8. The CRM of claim 7 , wherein the instructions to cause the apparatus to pass each frame of the sequence of frames through a depth estimation network comprise instructions to pass each frame of the sequence of frames through a monocular depth estimation network.
9. The CRM of claim 6 , wherein the camera pose comprises camera intrinsics.
10. The CRM of claim 6 , wherein the instructions to cause the apparatus to compare the resized camera view depth map and the estimated depth map to obtain the scale factor comprises instructions to cause the apparatus to perform a Procrustes analysis.
11. The CRM of claim 6 , wherein the apparatus is a server, and the frames are received from a remote device over a network in communication with the server.
12. A non-transitory computer generated medium (CRM) comprising instructions that, when executed by a processor of an apparatus, cause the apparatus to:
receive a sequence of frames of a scene captured by a camera and a camera pose of the camera;
pass each frame of the sequence of frames through a depth estimation network to obtain an estimated depth map;
generate a sparse depth map for each frame from the sequence of frames, each sparse depth map comprised of at least one 3D point;
render, from the sparse depth map and the camera pose, a camera view depth map representing the camera view; and
fit the camera view depth map to the estimated depth map to obtain a sparse depth map with an estimated metric scale by:
resize either or both of the camera view depth map and the estimated depth map;
compare the resized camera view depth map and the estimated depth map to obtain a disparity measure;
repeat the resizing and comparing until the disparity measure is reduced to a minimum;
obtain a scale factor that corresponds to the minimum disparity measure; and
apply the scale factor to the sparse depth map.
13. The CRM of claim 12 , wherein the estimated depth map comprises predicted per pixel depths.
14. The CRM of claim 13 , wherein the instructions to cause the apparatus to pass each frame of the sequence of frames through a depth estimation network comprise instructions to pass each frame of the sequence of frames through a monocular depth estimation network.
15. The CRM of claim 12 , wherein the camera pose comprises camera intrinsics.
16. The CRM of claim 12 , wherein the instructions to cause the apparatus to compare the resized camera view depth map and the estimated depth map to obtain the scale factor comprises instructions to cause the apparatus to perform a Procrustes analysis.
17. The CRM of claim 12 , wherein the apparatus is a server, and the frames and camera pose data are received from a remote device over a network in communication with the server.