Modeling indoor scenes based on digital images

1. A system comprising:

one or more computing devices;

memory; and

one or more computer programs stored in the memory for execution by the one or more computing devices, the one or more computer programs comprising instructions configured for:

receiving a request to produce a 3D model of an indoor scene based on multiple flat images of the indoor scene, wherein the multiple flat images of the indoor scene obey no more than a limited overlap requirement, are absent of depth information, and are taken from one or more viewpoints;

determining vanishing points in the multiple flat images;

receiving floor contour information, wherein the floor contour information consists of polygons, edges, and vertices determined based at least in part on the vanishing points;

reconstructing, using a geometric constraint matrix which encodes coordinate equalities among said vertices, 3D vertex positions of two or more floor plan parts based at least in part on the floor contour information;

determining a combined floor plan based at least in part on aligning the two or more floor plan parts, received length information, wherein the length information contains lengths of edges and groupings of edges of equal length, and received point correspondence information, wherein the point correspondence information indicates corresponding vertices among different images;

receiving floor plan outline information, wherein the floor plan outline information indicates walls in the indoor scene and was determined based at least in part on the combined floor plan; and

generating the 3D model of the indoor scene based at least in part on the floor plan outline information.

2. The system of claim 1 , wherein the one or more computer programs stored in the memory for execution by the one or more computing devices are further configured for determining vanishing points in the multiple flat images using line segment detection and classification according to cardinal axes X, Y and Z.

3. The system of claim 1 , wherein the one or more computer programs stored in the memory for execution by the one or more computing devices are further configured for aligning the two or more floor plan parts based at least in part on point correspondences among the two or more floor plan parts.

4. The system of claim 1 , wherein the one or more computer programs stored in the memory for execution by the one or more computing devices are further configured for aligning the two or more floor plan parts based at least in part on resolving rotational ambiguities among the two or more floor plan parts.

5. The system of claim 1 , wherein the one or more computer programs stored in the memory for execution by the one or more computing devices are further configured for aligning the two or more floor plan parts based at least in part on resolving scaling mismatches among the two or more floor plan parts.

6. The system of claim 1 , wherein the one or more computer programs stored in the memory for execution by the one or more computing devices are further configured for determining the geometric constraint matrix using connected component analysis.

7. The system of claim 1 , wherein the one or more computer programs stored in the memory for execution by the one or more computing devices are further configured for inferring a joint camera height for the multiple flat images based at least in part on an assumption that camera height is equal in each flat image of the multiple flat images.

8. A method comprising:

receiving, by one or more computing devices, a request to produce a 3D model of an indoor scene based on multiple flat images of the indoor scene, wherein the multiple flat images of the indoor scene obey no more than a limited overlap requirement, are absent of depth information, and are taken from one or more viewpoints;

determining, by the one or more computing devices, vanishing points in the multiple flat images;

receiving, by the one or more computing devices, floor contour information, wherein the floor contour information consists of polygons, edges and vertices determined based at least in part on the vanishing points;

reconstructing, by the one or more computing devices using a geometric constraint matrix that contains coordinate equalities among said vertices, 3D vertex positions of two or more floor plan parts based at least in part on the floor contour information;

determining, by the one or more computing devices, a combined floor plan based at least in part on the two or more floor plan parts, received length information, wherein the length information contains lengths of edges and groupings of edges of equal length, and received point correspondence information, wherein the point correspondence information indicates corresponding vertices among different images;

receiving, by the one or more computing devices, floor plan outline information, wherein the floor plan outline information indicates walls in the indoor scene and was determined based at least in part on the combined floor plan; and

generating, by the one or more computing devices, the 3D model of the indoor scene based at least in part on the floor plan outline information.

9. The method of claim 8 , further comprising displaying the 3D model on a graphical user interface.

10. The method of claim 8 , further comprising rendering the generated model in a head-mounted display.

11. The method of claim 8 , further comprising receiving floor plan trace information as the floor plan outline information, wherein the floor plan trace information is indicated by a user on a graphical user interface displaying the combined floor plan.

12. The method of claim 11 , wherein the graphical user interface on which the user indicated the floor plan trace information allows the user to opt to snap to vertex positions and introduce new vertices.

13. The method of claim 11 , wherein the graphical user interface on which the user indicated the floor plan trace information enforces right angles in the floor plan trace information.

14. The method of claim 8 , wherein a graphical user interface on which a user indicates the floor contour information, displays vanishing point crosshairs that show vanishing lines to guide the user to correctly draw floor contour information and indicate point correspondences in parts of a displayed image occluded by foreground objects.

15. The method of claim 8 , further comprising inferring a joint camera height for the multiple flat images based at least in part on an assumption that camera height is equal in each flat image of the multiple flat images.

16. A non-transitory computer readable medium storing instructions which, when executed by one or more computing devices, cause the one or more computing devices to perform a process of:

receiving a request to produce a 3D model of an indoor scene based on multiple flat images of the indoor scene, wherein the multiple flat images of the indoor scene obey no more than a limited overlap requirement, are absent of depth information, and are taken from one or more viewpoints;

determining vanishing points in the multiple flat images;

receiving floor contour information, wherein the floor contour information consists of polygons, edges and vertices determined based at least in part on the vanishing points;

reconstructing, by the one or more computing devices using a geometric constraint matrix that encodes coordinate equalities among of said vertices, 3D vertex positions of two or more floor plan parts based at least in part on the floor contour information;

determining a combined floor plan based at least in part on the two or more floor plan parts, received length information, wherein the length information contains lengths of edges and groupings of edges of equal length, and received point correspondence information, wherein the point correspondence information indicates corresponding vertices among different images;

receiving floor plan outline information, wherein the floor plan outline information indicates walls in the indoor scene and was determined based at least in part on the combined floor plan; and

generating the 3D model of the indoor scene based at least in part on the floor plan outline information.

17. The non-transitory computer readable medium of claim 16 , the process further comprising receiving additional vanishing lines and their classifications from a user.

18. The non-transitory computer readable medium of claim 16 , the process further comprising receiving the floor contour information with connected oriented and un-oriented edges.

19. The non-transitory computer readable medium of claim 16 , the process further comprising assembling the combined floor plan based at least in part on the two or more floor plan parts and the length information.

20. The non-transitory computer readable medium of claim 16 , the process further comprising inferring a joint camera height for the multiple flat images based at least in part on an assumption that camera height is equal in each flat image of the multiple flat images.

21. The non-transitory computer readable medium of claim 16 , the process further comprising receiving an indication of one or more openings in the indicated walls and generating the 3D model based at least in part on the one or more openings.