Image-based rendering of real spaces
Under an embodiment of the invention, an image capturing and processing system creates 3D image-based rendering (IBR) for real estate. The system provides image-based rendering of real property, the computer system including a user interface for visually presenting an image-based rendering of a real property to a user; and a processor to obtain two or more photorealistic viewpoints from ground truth image data capture locations; combine and process two or more instances of ground truth image data to create a plurality of synthesized viewpoints; and visually present a viewpoint in a virtual model of the real property on the user interface, the virtual model including photorealistic viewpoints and synthesized viewpoints.
1. A method, comprising:
receiving image data of a plurality of spaces in a property, the image data including a plurality of images captured from a plurality of viewpoints;
creating a plurality of panoramas of the plurality of spaces using the image data;
rendering a virtual model of a selected space among the plurality of spaces using the plurality of panoramas;
causing a device to display the virtual model with a first label indicating a location of the selected space; and
defining spatial boundaries of the plurality of spaces in the property using the image data, the plurality of spaces including a plurality of rooms in the property,
wherein the image data includes metadata associated with the plurality of images, the metadata indicating capture locations of the images,
wherein rendering the virtual model of the selected space includes rendering a 3D scene of the selected space using the plurality of panoramas, and
wherein rendering the 3D scene includes:
determining camera geometry using the plurality of images;
receiving a selected viewpoint in the selected space;
generating a point cloud of the 3D model;
determining a geometric proxy for the selected viewpoint using the determined camera geometry and the point cloud; and
generating the 3D model using the geometric proxy.
2. The method of claim 1 , further comprising:
calibrating the plurality of panoramas using the camera geometry,
wherein the camera geometry is determined by a first pass feature detection using the plurality of images.
3. The method of claim 1 , wherein the plurality of panoramas are grouped according to a panorama bundle adjustment heuristic.
4. The method of claim 3 , wherein the panorama bundle adjustment heuristic is initialized using the metadata, the metadata including one or both of GPS and WiFi positioning coordinates.
5. The method of claim 1 , further comprising:
generating the plurality of spatial boundaries of the plurality of spaces using the plurality of images, each of the spatial boundaries defining a parcel outline of one of the plurality of spaces;
annotating the plurality of images with second labels indicating capture locations of the plurality of images using the plurality of spatial boundaries; and
identifying the selected space among the plurality of spaces where a selected viewpoint is located using the second labels,
wherein causing the device to display the virtual model with the first label indicating the location of the selected space includes visually presenting the virtual model on a first portion of a user interface, a map overlay on a second portion of the user interface, and a text overlay on a third portion of the user interface, the map overlay indicating a position of the selected viewpoint in the selected space, the text overlay including the second label identifying the selected space in the property where the selected viewpoint is located.
6. The method of claim 1 , further comprising:
obtaining depth data of the plurality of spaces,
wherein creating the plurality of panoramas of the plurality of spaces includes generating a plurality of stitched panoramas from the plurality of images, the metadata, and the depth data.
7. The method of claim 6 wherein generating the plurality of stitched panoramas includes:
generating a panorama neighborhood graph of the property;
generating a panorama spanning tree of the panorama neighborhood graph; and
performing panorama bundle adjustment on the panorama spanning tree.
8. The method of claim 6 , rendering the virtual model includes registering image RGB data from the plurality of images with the depth data to generate a 3D point cloud.
9. The method of claim 1 , further comprising:
receiving the plurality of images from a smart phone.
10. A system, comprising:
a processor;
a memory storing non-transitory program commands, which, when executed by the processor, cause the processor to:
receive image data of a plurality of spaces in a property, the image data including a plurality of images captured from a plurality of viewpoints;
create a plurality of panoramas of the plurality of spaces using the image data;
render a virtual model of a selected space among the plurality of spaces using the plurality of panoramas;
cause a device to display the virtual model with a first label indicating a location of the selected space,
wherein the image data includes metadata associated with the plurality of images, the metadata indicating capture locations of the images,
wherein the program commands cause the processor to render the virtual model of the selected space by rendering a 3D scene of the selected space using the plurality of panoramas, and
wherein the program commands cause the processor to render the 3D scene by:
determining camera geometry using the plurality of images;
receiving a selected viewpoint in the selected space;
generating a point cloud of the 3D model;
determining a geometric proxy for the selected viewpoint using the determined camera geometry and the point cloud; and
generating the 3D model using the geometric proxy.
11. The system of claim 10 , wherein the program commands further cause the processor to:
obtain depth data of the plurality of spaces, and
wherein the program commands cause the processor to create the plurality of panoramas of the plurality of spaces by generating a plurality of stitched panoramas from the plurality of images, the metadata, and the depth data.
12. The system of claim 10 , wherein the program commands further cause the processor to:
generate a plurality of spatial boundaries of the plurality of spaces using the plurality of images, each of the spatial boundaries defining a parcel outline of one of the plurality of spaces;
annotate the plurality of images with second labels indicating capture locations of the plurality of images using the plurality of spatial boundaries; and
identify the selected space among the plurality of spaces where a selected viewpoint is located using the second labels,
wherein the program commands cause the processor to cause the device to display the virtual model with the first label indicating the location of the selected space by visually presenting the virtual model on a first portion of a user interface, a map overlay on a second portion of the user interface, and a text overlay on a third portion of the user interface, the map overlay indicating a position of the selected viewpoint in the selected space, the text overlay including the second label identifying the selected space in the property where the selected viewpoint is located.
13. A method, comprising:
receiving image data of a plurality of spaces in a property, the image data including a plurality of images captured from a plurality of viewpoints;
creating a plurality of panoramas of the plurality of spaces using the image data;
rendering a virtual model of a selected space among the plurality of spaces using the plurality of panoramas;
causing a device to display the virtual model with a first label indicating a location of the selected space; and
obtaining depth data of the plurality of spaces,
wherein creating the plurality of panoramas of the plurality of spaces includes generating a plurality of stitched panoramas from the plurality of images, the metadata, and the depth data, and
wherein generating the plurality of stitched panoramas includes:
generating a panorama neighborhood graph of the property;
generating a panorama spanning tree of the panorama neighborhood graph; and
performing panorama bundle adjustment on the panorama spanning tree.
14. The method of claim 13 , wherein the image data includes metadata associated with the plurality of images, the metadata indicating capture locations of the images, and
wherein rendering the virtual model of the selected space includes rendering a 3D scene of the selected space using the plurality of panoramas.
15. The method of claim 14 , further comprising:
obtaining depth data of the plurality of spaces,
wherein creating the plurality of panoramas of the plurality of spaces includes generating a plurality of stitched panoramas from the plurality of images, the metadata, and the depth data.
16. The method of claim 14 , further comprising:
defining spatial boundaries of the plurality of spaces in the property using the image data, the plurality of spaces including a plurality of rooms in the property.
17. The method of claim 16 , further comprising:
calibrating the plurality of panoramas using camera geometry,
wherein rendering the 3D scene includes:
determining the camera geometry using the plurality of images;
receiving a selected viewpoint in the selected space;
generating a point cloud of the 3D model;
determining a geometric proxy for the selected viewpoint using the determined camera geometry and the point cloud; and
generating the 3D model using the geometric proxy, and
wherein the camera geometry is determined by a first pass feature detection using the plurality of images.
18. The method of claim 13 , wherein the plurality of panoramas are grouped according to a panorama bundle adjustment heuristic.
19. The method of claim 18 , wherein the panorama bundle adjustment heuristic is initialized using the metadata, the metadata including one or both of GPS and WiFi positioning coordinates.