System for building a map and subsequent localization
SLAM systems are provided that utilize an artificial neural network to both map environments and locate positions within the environments. In some example embodiments, a sensor arrangement is used to map an environment. The sensor arrangement acquires sensor data from the various sensors and associates the sensor data, or data derived from the sensor data, with spatial regions in the environment. The sensor data may include image data and inertial measurement data that effectively describes the visual appearance of a spatial region at a particular location and orientation. This diverse sensor data may be fused into camera poses. The map of the environment includes camera poses organized by spatial region within the environment. Further, in these examples, an artificial neural network is adapted to the features of the environment by a transfer learning process using image data associated with camera poses.
1. A simultaneous localization and mapping (SLAM) system comprising:
a memory; and
at least one processor coupled to the memory and configured to:
receive sensor data descriptive of a plurality of spatial regions, the sensor data including image data;
train, using the image data, an artificial neural network to identify spatial regions of the plurality of spatial regions;
provide, to the artificial neural network, new sensor data descriptive of a spatial region of the plurality of spatial regions;
receive, from the artificial neural network, information identifying the spatial region of the plurality of spatial regions;
identify one or more keyframes associated with the spatial region; and
identify at least one camera pose using the one or more keyframes associated with the spatial region.
2. The SLAM system of claim 1 , wherein the sensor data includes one or more of three-dimensional image data, two-dimensional image data, and inertial measurement data.
3. The SLAM system of claim 1 , wherein the information identifying the spatial region of the plurality of spatial regions includes a list of spatial region identifiers and associated confidence levels.
4. The SLAM system of claim 1 , wherein the at least one processor is further configured to generate map data from the sensor data, the map data including one or more associations that associate the image data with one or more spatial regions of the plurality of spatial regions.
5. The SLAM system of claim 4 , wherein the image data includes the one or more keyframes, each of the one or more keyframes includes one or more keypoints, and the at least one processor is further configured to:
determine whether the new sensor data describes the one or more keypoints; and
identify the at least one camera pose by identifying one or more camera poses associated with the one or more keypoints in response to determining that the new sensor data describes the one or more keypoints.
6. The SLAM system of claim 4 , further comprising one or more sensors, wherein the at least one processor is further configured to:
receive a first message indicating that the one or more sensors are located in one spatial region of the plurality of spatial regions; and
generate an association of the one or more associations based on sensor data received after the first message.
7. The SLAM system of claim 6 , wherein the at least one processor is further configured to:
receive a second message requesting re-localization of the one or more sensors; and
provide the new sensor data to the artificial neural network in response to receiving the second message.
8. A method of mapping and localizing within an environment using a computing device, the method comprising:
receiving sensor data descriptive of a plurality of spatial regions, the sensor data including image data;
training, using the image data, an artificial neural network to identify spatial regions of the plurality of spatial regions;
providing new sensor data descriptive of a spatial region of the plurality of spatial regions to the artificial neural network;
receiving, from the artificial neural network, information identifying the spatial region of the plurality of spatial regions;
identifying one or more keyframes associated with the spatial region; and
identifying at least one camera pose using the one or more keyframes associated with the spatial region.
9. The method of claim 8 , wherein receiving the sensor data includes receiving one or more of three-dimensional image data, two-dimensional image data, and inertial measurement data.
10. The method of claim 8 , wherein receiving the information identifying the spatial region of the plurality of spatial regions includes receiving a list of spatial region identifiers and associated confidence levels.
11. The method of claim 8 , further comprising generating map data from the sensor data, the map data associating the sensor data with one or more spatial regions of the plurality of spatial regions and associating keyframes comprising keypoints with the one or more spatial regions.
12. The method of claim 11 , wherein the image data includes the one or more keyframes, each of the one or more keyframes includes one or more keypoints, and the method further comprises:
determining whether the new sensor data describes the one or more keypoints; and
identifying the at least one camera pose by identifying one or more camera poses associated with the one or more keypoints in response to determining that the new sensor data describes the one or more keypoints.
13. The method of claim 11 , further comprising:
receiving a first message indicating that one or more sensors are located in one spatial region of the plurality of spatial regions; and
generating an association between sensor data received after the first message and the one spatial region.
14. The method of claim 13 , further comprising:
receiving a second message requesting re-localization of the one or more sensors; and
providing the new sensor data to the artificial neural network in response to receiving the second message.
15. A non-transitory computer program product encoded with instructions that when executed by one or more processors cause a process for mapping and localizing within an environment using an artificial neural network to be carried out, the process comprising:
receiving sensor data descriptive of a plurality of spatial regions, the sensor data including image data;
training, using the image data, the artificial neural network to identify spatial regions of the plurality of spatial regions;
providing new sensor data descriptive of a spatial region of the plurality of spatial regions to the artificial neural network;
receiving information identifying the spatial region of the plurality of spatial regions from the artificial neural network;
identifying one or more keyframes associated with the spatial region; and
identifying at least one camera pose using the one or more keyframes associated with the spatial region.
16. The computer program product of claim 15 , wherein receiving the sensor data includes receiving one or more of three-dimensional image data, two-dimensional image data, and inertial measurement data.
17. The computer program product of claim 15 , wherein receiving the information identifying the spatial region of the plurality of spatial regions includes receiving a list of spatial region identifiers and associated confidence levels.
18. The computer program product of claim 15 , the process further comprising generating map data from the sensor data, the map data associating the sensor data with one or more spatial regions of the plurality of spatial regions.
19. The computer program product of claim 18 , wherein the image data includes the one or more keyframes, each of the one or more keyframes includes one or more keypoints, and the process further comprises:
determining whether the new sensor data describes the one or more keypoints; and
identifying the at least one camera pose by identifying one or more camera poses associated with the one or more keypoints in response to determining that the new sensor data describes the one or more keypoints.
20. The computer program product of claim 18 , the process further comprising:
receiving a first message indicating that one or more sensors are located in one spatial region of the plurality of spatial regions; and
generating an association between sensor data received after the first message and the one spatial region.
21. The computer program product of claim 20 , the process further comprising:
receiving a second message requesting re-localization of the one or more sensors; and
providing the new sensor data to the artificial neural network in response to receiving the second message.