Plane extraction and localization
Various implementations disclosed herein include devices, systems, and methods that localize a device based on detecting planes in depth data acquired by the device. For example, an example process may include detecting first plane data in first sensor data acquired by a sensor at a first viewpoint location in a physical environment, detecting second plane data in second sensor data acquired by the sensor at a second viewpoint location in the physical environment, determining that the first plane data and the second plane data correspond to a same plane based on comparing the first plane data with the second plane data, and determining a spatial transformation between the first viewpoint location and the second viewpoint location based on the first plane data and the second plane data.
1. A method comprising:
at a device having a processor and one or more sensors:
obtaining sensor data acquired by the one or more sensors, wherein at least one of the one or more sensors comprises a depth camera or a light intensity image camera from which the device is configured to obtain information about surfaces within an environment;
determining a first location of the device in a three-dimensional (3D) coordinate system of a physical environment for a first period of time based on motion data that is determined based on depth data, wherein the depth data is determined based on the sensor data acquired by the depth camera or the light intensity image camera;
estimating a first plane parameter of a first plane based on first plane data detected in a first set of sensor data acquired by the one or more sensors at a first viewpoint location in a physical environment, the first plane parameter estimated based on classifying a first set of 3D points of the first plane data into planes and estimating the first plane parameter based on the first set of 3D points, wherein the first plane parameter corresponds to 3D position or 3D orientation of the first plane;
estimating a second plane parameter of a second plane based on detecting second plane data in a second set of sensor data acquired by the one or more sensors at a second viewpoint location in the physical environment, the second plane parameter estimated based on classifying a second set of 3D points of the second plane data into planes and estimating the second plane parameter based on the second set of 3D points, wherein the second plane parameter corresponds to 3D position or 3D orientation of the second plane, and wherein the second viewpoint location is different than the first viewpoint location;
determining that the first plane and the second plane correspond to a same plane based on comparing the first plane parameter and the second plane parameter;
determining a spatial transformation between the first viewpoint location and the second viewpoint location based on determining that the first plane and the second plane correspond to the same plane;
determining localization data for tracking the location of the device in the 3D coordinate system based on: i) the motion data, ii) the spatial transformation between the first viewpoint location and the second viewpoint location, and iii) a distance from the device to the determined same plane; and
determining a second location of the device in the 3D coordinate system of the physical environment for a second period of time based on the localization data.
2. The method of claim 1 , wherein determining the spatial transformation comprises determining a motion constraint based on a first plane normal vector of the first plane data and second plane normal vector of the second plane data.
3. The method of claim 1 , wherein determining the spatial transformation comprises determining a second motion constraint based on a sensor-to-plane distance of the first plane data and a sensor-to-plane distance of the second plane data.
4. The method of claim 1 , wherein determining the spatial transformation comprises determining motion constraints based on:
a first plane normal vector of the first plane data and second plane normal vector of the second plane data;
a sensor-to-plane distance of the first plane data and a sensor-to-plane distance of the second plane data; and
covariance data of the plane normal vectors and the sensor-to-plane distances.
5. The method of claim 1 , wherein determining the spatial transformation comprises determining a motion based on motion data from an inertial measurement unit (IMU).
6. The method of claim 1 , wherein the sensor is a depth sensor, and wherein the first sensor data and second sensor data comprise a grid of depth values obtained via the depth sensor.
7. The method of claim 1 , wherein determining that the first plane and the second plane correspond to the same plane comprises determining that a direction of a first plane normal vector of the first plane data and a direction of a second plane normal vector of the second plane data are within a normal vector angle threshold.
8. The method of claim 7 , wherein determining that the first plane and the second plane correspond to the same plane further comprises determining a sensor-to plane distance in the first plane data and a sensor-to-plane distance in the second plane data are within a sensor-to-plane distance threshold.
9. The method of claim 1 , further comprising
identifying multiple planes represented in the first sensor data and the second sensor data; and
determining motion constraints based on the multiple planes,
wherein determining the spatial transformation is based on the motion constraints.
10. A device comprising:
one or more sensors;
a non-transitory computer-readable storage medium; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the device to perform operations comprising:
obtaining sensor data acquired by the one or more sensors, wherein at least one of the one or more sensors comprises a depth camera or a light intensity image camera from which the device is configured to obtain information about surfaces within an environment;
determining a first location of the device in a three-dimensional (3D) coordinate system of a physical environment for a first period of time based on motion data that is determined based on depth data, wherein the depth data is determined based on the sensor data acquired by the depth camera or the light intensity image camera;
estimating a first plane parameter of a first plane based on first plane data detected in a first set of sensor data acquired by the one or more sensors at a first viewpoint location in a physical environment, the first plane parameter estimated based on classifying a first set of 3D points of the first plane data into planes and estimating the first plane parameter based on the first set of 3D points, wherein the first plane parameter corresponds to 3D position or 3D orientation of the first plane;
estimating a second plane parameter of a second plane based on detecting second plane data in a second set of sensor data acquired by the one or more sensors at a second viewpoint location in the physical environment, the second plane parameter estimated based on classifying a second set of 3D points of the second plane data into planes and estimating the second plane parameter based on the second set of 3D points, wherein the second plane parameter corresponds to 3D position or 3D orientation of the second plane, and wherein the second viewpoint location is different than the first viewpoint location;
determining that the first plane and the second plane correspond to a same plane based on comparing the first plane parameter and the second plane parameter;
determining a spatial transformation between the first viewpoint location and the second viewpoint location based on determining that the first plane and the second plane correspond to the same plane;
determining localization data for tracking the location of the device in the 3D coordinate system based on: i) the motion data, ii) the spatial transformation between the first viewpoint location and the second viewpoint location, and iii) a distance from the device to the determined same plane; and
determining a second location of the device in the 3D coordinate system of the physical environment for a second period of time based on the localization data.
11. The device of claim 10 , wherein determining the spatial transformation comprises determining a motion constraint based on a first plane normal vector of the first plane data and second plane normal vector of the second plane data.
12. The device of claim 10 , wherein determining the spatial transformation comprises determining a second motion constraint based on a sensor-to-plane distance of the first plane data and a sensor-to-plane distance of the second plane data.
13. The device of claim 10 , wherein determining the spatial transformation comprises determining motion constraints based on:
a first plane normal vector of the first plane data and second plane normal vector of the second plane data;
a sensor-to-plane distance of the first plane data and a sensor-to-plane distance of the second plane data; and
covariance data of the plane normal vectors and the sensor-to-plane distances.
14. The device of claim 10 , wherein determining the spatial transformation comprises determining a motion based on motion data from an inertial measurement unit (IMU).
15. The device of claim 10 , wherein the first sensor data and second sensor data comprise a grid of depth values obtained via a depth sensor.
16. The device of claim 10 , wherein determining that the first plane and the second plane correspond to the same plane comprises:
determining that a direction of a first plane normal vector of the first plane data and a direction of a second plane normal vector of the second plane data are within a normal vector angle threshold; and
determining a sensor-to plane distance in the first plane data and a sensor-to-plane distance in the second plane data are within a sensor-to-plane distance threshold.
17. The device of claim 10 , wherein the operations further comprise:
identifying multiple planes represented in the first sensor data and the second sensor data; and
determining motion constraints based on the multiple planes,
wherein determining the spatial transformation is based on the motion constraints.
18. A non-transitory computer-readable storage medium, storing computer-executable program instructions on a device to perform operations comprising:
obtaining sensor data acquired by one or more sensors at the device, wherein at least one of the one or more sensors comprises a depth camera or a light intensity image camera from which the device is configured to obtain information about surfaces within an environment;
determining a first location of the device in a three-dimensional (3D) coordinate system of a physical environment for a first period of time based on motion data that is determined based on depth data, wherein the depth data is determined based on the sensor data acquired by the depth camera or the light intensity image camera;
estimating a first plane parameter of a first plane based on first plane data detected in a first set of sensor data acquired by the one or more sensors at a first viewpoint location in a physical environment, the first plane parameter estimated based on classifying a first set of 3D points of the first plane data into planes and estimating the first plane parameter based on the first set of 3D points, wherein the first plane parameter corresponds to 3D position or 3D orientation of the first plane;
estimating a second plane parameter of a second plane based on detecting second plane data in a second set of sensor data acquired by the one or more sensors at a second viewpoint location in the physical environment, the second plane parameter estimated based on classifying a second set of 3D points of the second plane data into planes and estimating the second plane parameter based on the second set of 3D points, wherein the second plane parameter corresponds to 3D position or 3D orientation of the second plane, and wherein the second viewpoint location is different than the first viewpoint location;
determining that the first plane and the second plane correspond to a same plane based on comparing the first plane parameter and the second plane parameter;
determining a spatial transformation between the first viewpoint location and the second viewpoint location based on determining that the first plane and the second plane correspond to the same plane;
determining localization data for tracking the location of the device in the 3D coordinate system based on: i) the motion data, ii) the spatial transformation between the first viewpoint location and the second viewpoint location, and iii) a distance from the device to the determined same plane; and
determining a second location of the device in the 3D coordinate system of the Physical environment for a second period of time based on the localization data.
19. The non-transitory computer-readable storage medium of claim 18 , wherein determining the spatial transformation comprises:
determining a motion constraint based on a first plane normal vector of the first plane data and second plane normal vector of the second plane data; or
determining a second motion constraint based on a sensor-to-plane distance of the first plane data and a sensor-to-plane distance of the second plane data.
20. The non-transitory computer-readable storage medium of claim 18 , wherein determining the spatial transformation comprises determining motion constraints based on:
a first plane normal vector of the first plane data and second plane normal vector of the second plane data;
a sensor-to-plane distance of the first plane data and a sensor-to-plane distance of the second plane data; and
covariance data of the plane normal vectors and the sensor-to-plane distances.
21. The method of claim 1 , wherein the second location is different than the first location in the 3D coordinate system of the physical environment.