Rendering depth-based three-dimensional model with integrated image frames
A system aligns a 3D model of an environment with image frames of the environment and generates a visualization interface that displays a portion of the 3D model and a corresponding image frame. The system receives LIDAR data collected in the environment and generates a 3D model based on the LIDAR data. For each image frame, the system aligns the image frame with the 3D model. After aligning the image frames with the 3D model, when the system presents a portion of the 3D model in an interface, it also presents an image frame that corresponds to the portion of the 3D model.
1. A method comprising:
receiving image frames collected by a mobile device as the mobile device is moved through an environment;
generating a 3D model representing the environment based on the received image frames;
aligning the image frames with the 3D model;
generating an interface displaying a portion of the 3D model;
receiving a selection of an object included within the displayed 3D model;
identifying an image frame corresponding to the selected object; and
modifying the generated interface to include the image frame displayed based on an angle of capture of the image frame.
2. The method of claim 1 , further comprising:
modifying the interface to include the image frame overlaid onto the 3D model at a location corresponding to the selected object.
3. The method of claim 2 , wherein the image frame is rendered within the 3D model at an angle perpendicular to the angle of capture of the image frame by the mobile device.
4. The method of claim 1 , further comprising:
receiving a selection of two endpoints of the object within 3D model displayed within the interface;
determining a distance between the two endpoints; and
displaying the determined distance within the interface.
5. The method of claim 1 , further comprising:
receiving a selection of two endpoints of the object within the image frame within the interface;
determining a distance between the two endpoints; and
displaying the determined distance within the interface.
6. The method of claim 1 , wherein the three dimensional (3D) model is generated by performing a simultaneous localization and mapping process on LIDAR data captured by the mobile device.
7. The method of claim 1 , wherein aligning the image frames with the 3D model further comprises:
determining a first set of features vectors associated with a plurality of points in the 3D model generated based on LIDAR data captured by the mobile device;
generating a second 3D model representing the environment based on the image frames;
determining a second set of feature vectors associated with a plurality of points in the second 3D model generated based on the image frames; and
mapping the plurality of points in the 3D model to the plurality of points in the second 3D model based on the first set of feature vectors and the second set of feature vectors.
8. The method of claim 1 , wherein aligning the image frames with the 3D model further comprises:
for each image frame;
determining a period of time associated with the image frame;
identifying a portion of LIDAR data captured by the mobile device associated with the period of time, the portion of the LIDAR data associated with a portion of the 3D model; and
storing an identification of the image frame in association with the identified portion of LIDAR data.
9. The method of claim 1 , wherein aligning the image frames with the 3D model further comprises:
extracting features associated with the 3D model;
comparing extracted features to annotations associated with one or more image frames; and
based on the comparison, storing an identification of an image frame in association with a portion of the 3D model, wherein one or more annotations associated with the image frame matches one or more features of the portion of the 3D model.
10. The method of claim 1 , further comprising:
receiving an interaction with the interface;
updating the interface to display a different portion of the 3D model according to the interaction; and
updating the interface to display a different image frame according to the interaction.
11. The method of claim 10 , wherein the interaction comprises a dragging action.
12. The method of claim 10 , wherein the interaction includes at least one of zooming in, zooming out, rotating, and shifting.
13. A non-transitory computer-readable storage medium storing executable instructions that, when executed by a hardware processor, cause the hardware processor to perform steps comprising:
receiving image frames collected by a mobile device as the mobile device is moved through an environment;
generating a 3D model representing the environment based on the received image frames;
aligning the image frames with the 3D model;
generating an interface displaying a portion of the 3D model;
receiving a selection of an object included within the displayed 3D model;
identifying an image frame corresponding to the selected object; and
modifying the generated interface to include the image frame displayed based on an angle of capture of the image frame.
14. The non-transitory computer-readable storage medium of claim 13 further storing executable instructions that, when executed by the hardware processor, further cause the hardware processor to perform steps comprising:
modifying the interface to include the image frame overlaid onto the 3D model at a location corresponding to the selected object.
15. The non-transitory computer-readable storage medium of claim 14 , wherein the image frame is rendered within the 3D model at an angle perpendicular to the angle of capture of the image frame by the mobile device.
16. The non-transitory computer-readable storage medium of claim 13 further storing executable instructions that, when executed by the hardware processor, further cause the hardware processor to perform steps comprising:
receiving a selection of two endpoints of the object within 3D model displayed within the interface;
determining a distance between the two endpoints; and
displaying the determined distance within the interface.
17. The non-transitory computer-readable storage medium of claim 13 further storing executable instructions that, when executed by the hardware processor, further cause the hardware processor to perform steps comprising:
receiving a selection of two endpoints of the object within the image frame within the interface;
determining a distance between the two endpoints; and
displaying the determined distance within the interface.
18. The non-transitory computer-readable storage medium of claim 13 , wherein the three dimensional (3D) model is generated by performing a simultaneous localization and mapping process on LIDAR data captured by the mobile device.
19. A system comprising:
a processor; and
a non-transitory computer readable storage medium comprising computer program instructions that when executed by the processor, cause the processor to:
receive image frames collected by a mobile device as the mobile device is moved through an environment;
generate a 3D model representing the environment based on the received image frames;
align the image frames with the 3D model;
generate an interface displaying a portion of the 3D model;
receive a selection of an object included within the displayed 3D model;
identify an image frame corresponding to the selected object; and
modify the generated interface to include the image frame displayed based on an angle of capture of the image frame.
20. The system of claim 19 , wherein the computer program instructions cause the processor to modify the interface to include the image frame overlaid onto the 3D model at a location corresponding to the selected object.