Mobile device localization in complex, three-dimensional scenes
The present embodiments relate to localizing a mobile device in a complex, three-dimensional scene. By way of introduction, the present embodiments described below include apparatuses and methods for using multiple, independent pose estimations to increase the accuracy of a single, resulting pose estimation. The present embodiments increase the amount of input data by windowing a single depth image, using multiple depth images from the same sensor, and/or using multiple depth image from different sensors. The resulting pose estimation uses the input data with a multi-window model, a multi-shot model, a multi-sensor model, or a combination thereof to accurately estimate the pose of a mobile device.
1. A method of estimating a pose of a mobile device in a three-dimensional scene, the method comprising:
receiving, by a processor of the mobile device, a plurality of depth data measurements, the depth data measurements indicative of a depth from the mobile device to the three-dimensional scene;
estimating, based on a first depth measurement of the plurality of depth data measurements, a first pose of the mobile device with respect to the three-dimensional scene;
estimating, based on a second depth measurement of the plurality of depth data measurements, a second pose of the mobile device with respect to the three-dimensional scene;
estimating, with a pose model based on the first pose and the second pose, a third pose of the mobile device with respect to the three-dimensional scene; and
providing, based on the third pose, an output to a user.
2. The method of claim 1 , wherein the plurality of depth data measurements comprise measurements of a same sensor, and wherein the pose model is a motion model based on movement of the same sensor between measurements.
3. The method of claim 2 , wherein the motion model is a linear motion model based on sensor velocity measurements and corresponding time measurements.
4. The method of claim 2 , wherein the motion model is a learned motion model, wherein a plurality of possible trajectories of the sensor are determined by the plurality of depth data measurements, the first pose and the second pose.
5. The method of claim 1 , wherein the plurality of depth data measurements comprise measurements of different sensors, and wherein the pose model is a spatial model based on known spatial relationships between the different sensors.
6. The method of claim 1 , wherein the plurality of depth data measurements comprise a plurality of measurements of each of a plurality of different sensors, and wherein the pose model includes a motion model based on movement of each sensor between measurements by the sensor and a spatial model based on known spatial relationships between the different sensors.
7. The method of claim 1 , wherein the plurality of depth data measurements are partitioned from a single sensor measurement, and wherein the pose model comprises combining the first pose and the second pose into the third pose.
8. The method of claim 7 , wherein the combining comprises a weighted average, mode finding or semantic mapping.
9. The method of claim 1 , wherein the plurality of depth data measurements comprise measurements of a same sensor, and wherein a location of the mobile device for the second depth measurement is specified based on the first pose of the mobile device.
10. The method of claim 1 , wherein the three-dimensional scene comprises a three-dimensional apparatus, and wherein the output to the user is a component of the three-dimensional apparatus.
11. The method of claim 1 , wherein the output to the user comprises initialization or tracking in an augmented reality scenario.
12. A system for determining a pose of a mobile device in a three-dimensional scene, the system comprising:
a memory configured to store a plurality of depth data measurements, the depth data measurements indicative of a depth from the mobile device to the three-dimensional scene; and
a processor configured to:
receive the plurality of depth data measurements;
estimate, based on a first depth measurement of the plurality of depth data measurements, a first pose of the mobile device with respect to the three-dimensional scene;
estimate, based on a second depth measurement of the plurality of depth data measurements, the second depth measurement different from the first depth measurement of the plurality of depth data measurements, a second pose of the mobile device with respect to the three-dimensional scene;
estimate, with a pose model based on the first pose and the second pose, a third pose of the mobile device with respect to the three-dimensional scene; and
provide, based on the third pose, an output to a user.
13. The system of claim 12 , wherein a dynamic model comprises one or more of:
a multi-window fusion model based on partitioning sensor measurements into a plurality of windows for each sensor measurement;
a multi-shot fusion model based on sensor movement between sensor measurements; and
a multi-sensor fusion model based on known spatial relationships between different sensors.
14. The system of claim 12 , further comprising:
a sensor configured to capture the plurality of depth data measurements.
15. The system of claim 14 , wherein the sensor is a 2.5D sensor or a depth sensor.
16. The system of claim 12 , further comprising:
a display configured to provide the output to a user based on the estimated pose.