Capturing and aligning three-dimensional scenes
Systems and methods for building a three-dimensional composite scene are disclosed. Certain embodiments of the systems and methods may include the use of a three-dimensional capture device that captures a plurality of three-dimensional images of an environment. Some embodiments may further include elements concerning aligning and/or mapping the captured images. Various embodiments may further include elements concerning reconstructing the environment from which the images were captured. The methods disclosed herein may be performed by a program embodied on a non-transitory computer-readable storage medium when executed the program is executed a processor.
1. A device, comprising:
a processor, and
a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, comprising:
receiving three-dimensional data of an environment captured via a three-dimensional capture device of the device;
determining an alignment between the three-dimensional data and a three-dimensional model of the environment;
based on the alignment, determining a current position of the device relative to a first location on or within the three-dimensional model that is associated with an auxiliary data object, wherein the auxiliary data object comprises visual data capable of being presented on a display of the device; and
based on the current position of the device relative to the first location and a current viewpoint of the environment as viewed on or through the display, determining a perspective for displaying the auxiliary data object via the display that spatially aligns the auxiliary data object with a second location in the environment that corresponds to the first location on or within the three-dimensional model.
2. The device of claim 1 , wherein the operations further comprise:
displaying the auxiliary data object via the display based on the perspective.
3. The device of claim 1 , wherein the determining the alignment comprises determining the alignment by matching first spatial features of the three-dimensional data with second spatial features of the three-dimensional model.
4. The device of claim 1 , wherein the current viewpoint of the environment comprises a viewpoint of a user of the device relative to the environment and the display, and wherein the operations further comprise:
determining the current viewpoint based on image data captured of the user's face.
5. The device of claim 2 , wherein the displaying comprises overlaying the auxiliary data object over a live image of the environment using depth-based occlusion.
6. The device of claim 2 , wherein the operations further comprise:
generating a three-dimensional rendering comprising the auxiliary data object from the perspective by combining the auxiliary data object with the three-dimensional model at the first location on or within the three dimensional-model, and wherein the displaying comprises displaying the three-dimensional rendering.
7. The device of claim 2 , wherein the operations further comprise:
generating another three-dimensional model of the environment based on the three-dimensional data; and
generating a three-dimensional rendering comprising the auxiliary data object from the perspective by combining the other three-dimensional model with the auxiliary data object, and wherein the displaying comprises displaying the three-dimensional rendering.
8. The device of claim 1 , wherein the three-dimensional model of the environment is generated at a substantially same time as the the determining the alignment between the three-dimensional data of the environment and the three-dimensional model.
9. The device of claim 1 , wherein the three-dimensional model of the environment is generated and the auxiliary data is captured at a substantially same time as the the determining the alignment between the three-dimensional data of the environment and the three-dimensional model.
10. A method, comprising:
receiving, by a system comprising a processor, three-dimensional data captured of an environment by a three-dimensional capture device;
determining, by the system, an alignment between the three-dimensional data and a three-dimensional reconstruction of the environment;
based on the alignment, determining, by the system, a current position of the three-dimensional capture device relative to a first location on or within the three-dimensional reconstruction that is associated with auxiliary data, wherein the auxiliary data comprises visual data capable of being presented on a display; and
based on the current position of the three-dimensional capture device relative to the first location and a current viewpoint of the environment as viewed on or through the display, determining, by the system, a position for displaying the auxiliary data via the display that spatially aligns the auxiliary data with a second location in the environment that corresponds to the first location on or within the three-dimensional reconstruction.
11. The method of claim 10 , further comprising:
generating, by the system, a three-dimensional rendering comprising the auxiliary data at the position.
12. The method of claim 10 , further comprising:
displaying, by the system, the auxiliary data via the display.
13. The method of claim 12 , wherein the display comprises a transparent or partially transparent display and wherein the displaying comprises overlaying the auxiliary data object over a live view of the environment viewed through the display.
14. The method of claim 10 , wherein the current viewpoint of the environment comprises a viewpoint of a user relative to the environment and the display, the method further comprising:
receiving, by the system, image data of the user's face; and
determining, by the system, the current viewpoint based on the image data.
15. The method of claim 10 , wherein the determining the alignment comprises determining the alignment by matching first image or first spatial features of the live three-dimensional data with second image or second spatial features of the three-dimensional reconstruction.
16. The method of claim 11 , wherein the generating the three-dimensional rendering comprises combining the auxiliary data with the three-dimensional data.
17. The method of claim 11 , wherein the generating the three-dimensional rendering comprises combining the auxiliary data with the three-dimensional reconstruction.
18. The method of claim 11 , further comprising:
generating, by the system, another three-dimensional representation of the environment based on the three-dimensional data, wherein the generating the three-dimensional rendering comprises combining the other three-dimensional representation with the auxiliary data to.
19. The method of claim 10 , further comprising:
generating, by the system, the three-dimensional reconstruction at a substantially same time as the determining the alignment.
20. The method of claim 19 , further comprising:
capturing, by the system, the auxiliary data at a substantially same time as the generating and the determining the alignment.
21. The method of claim 10 , further comprising:
capturing, by the system, the auxiliary data via a sensor.
22. The method of claim 21 , wherein the sensor comprises a camera configured to capture surface information associated with the three-dimensional reconstruction.
23. A non-transitory computer-readable storage medium comprising computer-readable instructions that, in response to execution, cause a computing system to perform operations, comprising:
capturing first three-dimensional data of an environment using a three-dimensional capture device;
aligning the first three-dimensional data with a three-dimensional model of the environment generated based on second three-dimensional data of the environment, wherein the three-dimensional model is associated with auxiliary data that is spatially aligned with a first location on or within the three-dimensional model;
determining a current position of the three-dimensional capture device relative to the first position on or within the three-dimensional model based on the aligning; and
based on the current position of the three-dimensional capture device relative to the first location and a current viewpoint of the environment relative to a display, determining a perspective for displaying the auxiliary data object via the display that spatially aligns the auxiliary data object with a second location in the environment that corresponds to the first location on or within the three-dimensional model.
24. The non-transitory computer-readable storage medium system of claim 23 , wherein the aligning comprises matching first position information for one or more first features of the first three-dimensional data with second position information for one or more second features of the three-dimensional model.
25. The non-transitory computer-readable storage medium system of claim 23 , wherein the operations further comprise:
generating a three-dimensional rendering comprising the auxiliary data with the perspective; and
displaying the three-dimensional rendering via the display.
26. The device of claim 1 , wherein the determining the alignment comprises determining the alignment by matching first image features of the live three-dimensional data of the environment with second image features of the three-dimensional model.
27. The device of claim 1 , wherein the auxiliary data object comprises a visual mark or label identifying an object or location in the environment.
28. The device of claim 1 , wherein the auxiliary data object comprises an image or a graphical three-dimensional data object.
29. The device of claim 1 , wherein the auxiliary data object comprises a computer-aided design model of the environment.
30. The device of claim 2 , wherein the display comprises a transparent or partially transparent display and wherein the displaying comprises overlaying the auxiliary data object over a live view of the environment viewed through the display.