Methodology for 3D scene reconstruction from 2D image sequences
View Patent ↗The present invention is directed to a system and method for interactive and iterative reconstruction in a manner that helps to reduce computational requirements by generating a model from a subset of the available data, and then refining that model using additional data. Example embodiments directed to scene reconstruction, reconstruct a 3D scene from a plurality of 2D images of that scene by first generating a model of the 3D scene from a subset a of the 2D images. The model can then be refined using specific characteristics of each image in the subset that are calculated using the other images in the subset. The model is further refined using images not in the original subset.
1. A method of scene reconstruction comprising:
obtaining a plurality of images of a scene representing at least two perspective views of said scene;
generating a 3-dimensional model of said scene using a first subset of said plurality of images, wherein said generating comprises selecting one or more vertices of said 3-dimensional model;
calculating a perspective for each said image in said first subset based on one or more of said one or more selected vertices; and
refining said model based on one or more of said calculated perspectives and a second subset of said images, wherein said second subset includes at least one image not in said first subset.
2. The method of claim 1 further comprising:
refining said scene using said refined model.
3. The method of claim 1 wherein said first subset comprises:
a first image captured with a first camera view; and
a second image captured with a second camera view; and
wherein said second subset comprises:
a third image captured with a third camera view, wherein said third view is between said first view and said second view.
4. The method of claim 1 wherein said refining comprises iteratively adding one or more images from said plurality of images, not already in said second subset, to said second subset, and refining said scene using said second subset.
5. The method of claim 1 wherein user input is used to select an order of said plurality of images used in said scene reconstruction.
6. The method of claim 1 further comprising:
determining whether said refining achieves a sufficient quality level.
7. The method of claim 6 wherein said refining is done when said sufficient quality level has not been achieved.
8. The method of claim 1 wherein said generating comprises a user selecting a subset of vertices from a plurality of vertices within at least one of said first subset of images, and refining said user selected subset of vertices.
9. The method of claim 1 wherein said generation further comprises automated selection of at least one of said first subset of images based on motion analysis.
10. The method of claim 1 wherein said selection is based on a user's knowledge of said scene acquired from a graphical user interface.
11. The method of claim 1 , further comprising:
employing a process of resectioning to calculate camera positions for said images of said second subset.
12. The method of claim 1 , further comprising:
using said images of said first subset as key frames; and
using said images of said second subset as intermediate frames.
13. The method of claim 1 further comprising:
selecting, for one or more of said calculated perspectives, one or more visible vertices and one or more non-visible vertices from said one or more selected vertices; and
calculating a perspective for each image in said second subset of said plurality of images based on said one or more visible vertices.
14. The method of claim 1 wherein said selecting comprises selecting said one or more vertices by a user.
15. A non-transitory, machine-readable storage medium comprising instructions stored therein, the instructions executable by one or more processors to facilitate performing a method, said method comprising:
obtaining a plurality of images of a scene representing at least two perspective views of said scene;
generating a 3-dimensional model of said scene using a first subset of said plurality of images, wherein said generating comprises selecting one or more vertices of said 3-dimensional model;
calculating a perspective for each said image in said first subset based on one or more of said one or more selected vertices; and
refining said model based on one or more of said calculated perspectives and a second subset of said images, wherein said second subset includes at least one image not in said first subset.
16. The non-transitory machine-readable medium of claim 15 further comprising:
selecting, for one or more of said calculated perspectives, one or more visible vertices and one or more non-visible vertices from said one or more selected vertices; and
calculating a perspective for each image in said second subset of said plurality of images based on said one or more visible vertices.
17. The non-transitory machine-readable medium of claim 15 wherein said selecting comprises selecting said one or more vertices by a user.
18. A system, comprising:
a memory storing executable instructions;
one or more processors operatively coupled to said memory to execute the executable instructions from the memory, the one or more processors being configured to:
obtain a plurality of images of a scene representing at least two perspective views of said scene;
generate a 3-dimensional model of said scene using a first subset of said plurality of images, wherein said one or more processors are configured to select one or more vertices of said 3-dimensional model;
calculate a perspective for each said image in said first subset based on one or more of said one or more selected vertices; and
refine said model based on one or more of said calculated perspectives and a second subset of said images, wherein said second subset includes at least one image not in said first subset.
19. The system of claim 18 , said one or more processors being further configured to:
select, for one or more of said calculated perspectives, one or more visible vertices and one or more non-visible vertices from said one or more selected vertices; and
calculate a perspective for each image in said second subset of said plurality of images based on said one or more visible vertices.
20. The system of claim 18 wherein said one or more processors are configured to select said one or more vertices in response to a selection made by a user.