Methods and systems for detecting and combining structural features in 3D reconstruction
A method for forming a reconstructed 3D mesh includes receiving a set of captured depth maps associated with a scene, performing an initial camera pose alignment associated with the set of captured depth maps, and overlaying the set of captured depth maps in a reference frame. The method also includes detecting one or more shapes in the overlaid set of captured depth maps and updating the initial camera pose alignment to provide a shape-aware camera pose alignment. The method further includes performing shape-aware volumetric fusion and forming the reconstructed 3D mesh associated with the scene.
1. A method of detecting a shape present in a scene, the method comprising:
determining, using one or more processors and a plurality of point normals, a vertical direction associated with a point cloud including a plurality of captured depth maps;
forming, using the one or more processors, a virtual plane orthogonal to the vertical direction;
projecting, using the one or more processors, the points of the point cloud onto the virtual plane;
calculating, using the one or more processors, projection statistics for the points of the point cloud;
detecting, using the one or more processors, one or more lines from the calculated projection statistics, the one or more lines being associated with vertical walls; and
detecting, using the one or more processors, the shape present in the scene from the projection statistics and the one or more detected lines.
2. The method of claim 1 further comprising determining dimensions and positions of the detected shape.
3. The method of claim 1 wherein determining the vertical direction comprises, for a plurality of pixels in the point cloud:
determining a plurality of horizontal planes defined, for each of the plurality of pixels, by neighboring pixels to each of the plurality of pixels; and
computing a vector normal to the plurality of horizontal planes.
4. The method of claim 1 wherein the projection statistics comprise a number of points of the point cloud projected onto a predetermined x,y location in the virtual plane.
5. The method of claim 4 wherein the projection statistics comprise a distribution of point normals for the points of the point cloud projected onto the predetermined x,y location in the virtual plane.
6. The method of claim 4 wherein the projection statistics comprise an initial height of the points of the point cloud projected onto the predetermined x,y location in the virtual plane.
7. The method of claim 1 wherein the detected shape present in the scene comprises a wall corner.
8. The method of claim 1 wherein the detected shape present in the scene comprises a door frame.
9. The method of claim 1 wherein the plurality of captured depth maps are each obtained from different positions with respect to a scene.
10. The method of claim 1 wherein the plurality of captured depth maps are each obtained from a single position with respect to a scene at different times.
11. The method of claim 1 further comprising forming the point cloud by overlaying the plurality of captured depth maps in a reference frame.
12. The method of claim 1 wherein the vertical direction is aligned with a gravity vector.
13. The method of claim 1 wherein determining the vertical direction comprises:
estimating the vertical direction; and
refining the estimated vertical direction.
14. The method of claim 1 wherein projecting the points of the point cloud onto the virtual plane comprises forming a two-dimensional data set.