High-speed and tunable scene reconstruction systems and methods using stereo imagery
A tunable and iterative stereo mapping technique is provided, capable of identifying disparities at or substantially faster than real-time (e.g., frame-rate of 120 Hz). The method includes identifying a plurality of points in an image, determining disparity values for each of the points in the image and generating a piece-wise planar mesh based on the points and their respective disparity values. A disparity interpolation can be performed on candidate planes using estimated plane parameters for the candidate planes and a disparity image can be generated having a plurality of regions based on the disparity interpolation. Multiple iterations can be performed until the image is reconstructed with an appropriate resolution based on predetermined thresholds. The thresholds can be modified to provide a tunable system by changing the threshold values to either increase a resolution of a final reconstructed image and/or increase a computation speed of the tunable and iterative stereo mapping technique.
1. A method comprising:
identifying a plurality of points in a first image;
determining disparity values for each of the plurality of points in the first image;
generating a piece-wise planar mesh based on the plurality of points and their respective disparity values, wherein the piece-wise planar mesh is formed from a plurality of candidate planes;
using estimated plane parameters for each of the plurality of candidate planes to perform disparity interpolation on the plurality of candidate planes to generate a plurality of interpolated regions;
generating a disparity image from the plurality of interpolated regions, wherein each of the plurality of interpolated regions includes interpolated disparity values for one or more of the plurality of points in a respective interpolated region;
determining a cost value for each of the plurality of interpolated regions based on their respective interpolated disparity values; and
comparing the cost value for each of the plurality of interpolated regions to a cost threshold.
2. The method of claim 1 , further comprising validating one or more of the plurality of interpolated regions when the respective cost value is less than the cost threshold.
3. The method of claim 1 , further comprising invalidating one or more of the plurality of interpolated regions when the respective cost value is greater than or equal to the cost threshold.
4. The method of claim 1 , further comprising determining a set of support points for each of the plurality of points in the first image.
5. The method of claim 4 , further comprising determining a concatenation value for each of the support points.
6. The method of claim 4 , further comprising generating the piece-wise planar mesh using one or more of the support points as vertices.
7. The method of claim 1 , further comprising identifying triangular regions from the plurality of candidate planes and identifying 3D parameters for each of the triangular regions.
8. The method of claim 1 , further comprising generating a lookup table for each of the candidate planes, wherein the lookup table includes plane coefficients for each pixel in a respective one of the plurality of candidate planes.
9. The method of claim 1 , further comprising determining a grid size for each of the plurality of interpolated regions based on the respective cost value of the regions.
10. The method of claim 1 , further comprising reducing a grid size of each of the plurality of interpolated regions on each subsequent iteration.
11. The method of claim 3 , further comprising resampling invalidated regions.
12. The method of claim 11 , further comprising identifying a plurality of points in a first region of the first image, wherein the first region has been invalidated.
13. The method of claim 12 , further comprising determining disparity values for each of the plurality of points in the first region and generating a piece-wise planar mesh based on the plurality of points and their respective disparity values through Delaunay triangulation.
14. The method of claim 13 , further comprising performing disparity interpolation on the plurality of candidate planes using the estimated plane parameters for each of the plurality of candidate planes and generating a disparity image having a plurality of sub-regions in the first region based on the disparity interpolation.
15. The method of claim 14 , further comprising determining a cost value for each of the plurality of sub-regions based on their respective disparity values and comparing the cost value for each of the plurality of sub-regions to a cost threshold.
16. The method of claim 1 , further comprising generating the piece-wise planar mesh based on the plurality of points and their respective disparity values through Delaunay triangulation.
17. A method comprising:
identifying a plurality of points in a first image and a second image that correspond to sparse salient features;
determining disparity values for each of the plurality of points in the first image;
generating a piece-wise planar mesh based on the plurality of points and their respective disparity values, wherein the piece-wise planar mesh is formed from a plurality of candidate planes;
using estimated plane parameters for each of the plurality of candidate planes to perform disparity interpolation on the plurality of candidate planes to generate a plurality of interpolated regions;
generating a disparity image from the plurality of interpolated regions, wherein each of the plurality of interpolated regions includes interpolated disparity values for one or more of the plurality of points in a respective interpolated region;
determining a cost value for each of the plurality of interpolated regions based on their respective interpolated disparity values; and
comparing the cost value for each of the plurality of interpolated regions to a cost threshold.
18. The method of claim 17 , further comprising mapping the plurality of points in the first image to the plurality of points in the second image using their respective epipolar lines.
19. The method of claim 18 , further comprising identifying mapping points between the first and second images using an epipolar search.
20. The method of claim 17 , further comprising performing Census window-based mapping between related regions in the first image and a second image.
21. The method of claim 20 , wherein the cost value for each region corresponds to a disparity between the first and second images.