Systems and methods for augmented reality preparation, processing, and application
Various of the disclosed embodiments provide systems and methods for acquiring and applying a depth determination of an environment in e.g., various augmented reality applications. A user may passively or actively scan a device (e.g., a tablet device, a mobile phone device, etc.) about the environment acquiring depth data for various regions. The system may integrate these scans into an internal three-dimensional model. This model may then be used in conjunction with subsequent data acquisitions to determine a device's location and orientation within the environment with high fidelity. In some embodiments, these determinations may be accomplished in real-time or near-real-time. Using the high-fidelity orientation and position determination, various augmented reality applications may then be possible using the same device used to acquire the depth data or a new device.
1. A device to facilitate an augmented reality experience, the device comprising:
a depth sensor;
at least one processor;
at least one memory comprising instructions configured to cause the at least one processor to cause the device to perform a method comprising:
receiving depth frame data acquired using the depth sensor, the depth frame data comprising distances from the depth sensor to surfaces in a real-world environment;
determining a final pose, wherein determining the final pose comprises iteratively performing, for at least two iterations, a correspondence identification process and a pose determination process, wherein
the correspondence identification process comprises:
determining a plurality of correspondences between the frame data transformed using a pose and model data of the real-world environment, and wherein
the pose determination process comprises:
iteratively refining an intermediate pose determination, at least in part, by pruning candidate poses within a search space and considering poses from progressively smaller neighborhoods;
and
making the final pose available to an augmented reality application.
2. The device of claim 1 , wherein the pose determination process comprises a Pose Search Algorithm (PSA) and iteratively performing, for at least two iterations, a correspondence identification process and a pose determination process comprises performing an Expectation Maximization (EM) process.
3. The device of claim 2 , wherein the pose determination process generates:
a Bayesian belief probability distribution associated with a given time; and
a pose corresponding to a transform.
4. The device of claim 2 , wherein the PSA comprises a Scaling Series algorithm.
5. The device of claim 1 , wherein the correspondence identification process comprises a likelihood field model with KD-trees.
6. The device of claim 1 , wherein the correspondence identification process comprises an independent beam model.
7. The device of claim 1 , wherein determining the final pose further comprises:
determining a first pose from a first iteration of the correspondence identification process and the pose determination process;
determining a second pose from a second successive iteration of the correspondence identification process and the pose determination process;
determining a difference between the first pose and the second pose in meters-and-radians;
determining that the difference is beneath a threshold; and
outputting the second pose as the final pose.
8. A computer-implemented method comprising:
receiving depth frame data acquired using a depth sensor associated with an augmented reality device, the depth frame data comprising distances from the depth sensor to surfaces in a real-world environment;
determining a final pose, wherein determining the final pose comprises iteratively performing, for at least two iterations, a correspondence identification process and a pose determination process, wherein
the correspondence identification process comprises:
determining a plurality of correspondences between the frame data transformed using a pose and model data of the real-world environment, and wherein
the pose determination process comprises:
iteratively refining an intermediate pose determination, at least in part, by pruning candidate poses within a search space and considering poses from progressively smaller neighborhoods; and
making the final pose available to an augmented reality application.
9. The computer-implemented method of claim 8 , wherein the pose determination process comprises a Pose Search Algorithm (PSA) and iteratively performing, for at least two iterations, a correspondence identification process and a pose determination process comprises performing an Expectation Maximization (EM) process.
10. The computer-implemented method of claim 9 , wherein the pose determination process generates:
a Bayesian belief probability distribution associated with a given time; and
a pose corresponding to a transform.
11. The computer-implemented method of claim 9 , wherein the PSA comprises a Scaling Series algorithm.
12. The computer-implemented method of claim 8 , wherein the correspondence identification process comprises a likelihood field model with KD-trees.
13. The computer-implemented method of claim 8 , wherein the correspondence identification process comprises an independent beam model.
14. The computer-implemented method of claim 8 , wherein determining the final pose further comprises:
determining a first pose from a first iteration of the correspondence identification process and the pose determination process;
determining a second pose from a second successive iteration of the correspondence identification process and the pose determination process;
determining a difference between the first pose and the second pose in meters-and-radians;
determining that the difference is beneath a threshold; and
outputting the second pose as the final pose.
15. A non-transitory computer-readable medium comprising instructions configured to cause a computer system to perform a method, comprising:
receiving depth frame data acquired using a depth sensor associated with an augmented reality device, the depth frame data comprising distances from the depth sensor to surfaces in a real-world environment;
determining a final pose based, wherein determining the final pose comprises iteratively performing, for at least two iterations, a correspondence identification process and a pose determination process, wherein
the correspondence identification process comprises:
determining a plurality of correspondences between the frame data transformed using a pose and model data of the real-world environment, and wherein
the pose determination process comprises:
iteratively refining an intermediate pose determination, at least in part, by pruning candidate poses within a search space and considering poses from progressively smaller neighborhoods; and
making the final pose available to an augmented reality application.
16. The non-transitory computer-readable medium of claim 15 , wherein the pose determination process comprises a Pose Search Algorithm (PSA) and iteratively performing, for at least two iterations, a correspondence identification process and a pose determination process comprises performing an Expectation Maximization (EM) process.
17. The non-transitory computer-readable medium of claim 16 , wherein the pose determination process generates:
a Bayesian belief probability distribution associated with a given time; and
a pose corresponding to a transform.
18. The non-transitory computer-readable medium of claim 16 , wherein the PSA comprises a Scaling Series algorithm.
19. The non-transitory computer-readable medium of claim 15 , wherein the correspondence identification process comprises a likelihood field model with KD-trees.
20. The non-transitory computer-readable medium of claim 15 , wherein the correspondence identification process comprises an independent beam model.
21. The non-transitory computer-readable medium of claim 15 , wherein determining the final pose further comprises:
determining a first pose from a first iteration of the correspondence identification process and the pose determination process;
determining a second pose from a second successive iteration of the correspondence identification process and the pose determination process;
determining a difference between the first pose and the second pose in meters-and-radians;
determining that the difference is beneath a threshold; and
outputting the second pose as the final pose.