Incremental map building using learnable features and descriptors
A method for localization performed by an agent includes receiving a query image of a current environment of the agent captured by a sensor integrated with the agent. The method also includes receiving a target image comprising a first set of keypoints matching a second set of keypoints of the query image. The first set of keypoints may be generated based on a task specified for the agent. The method still further includes determining a current location based on the target image.
1. A method for localization performed by a vehicle, comprising:
capturing, via one or more sensors integrated with the vehicle, a query image of a current environment of the vehicle, the query image being a two-dimensional (2D) image;
identifying a target image, from a plurality of images associated with a three-dimensional (3D) map of the current environment, comprising a first set of keypoints that match a second set of keypoints of the query image, each image of the plurality of images associated with a respective set of keypoints that are labelled via a keypoint model trained for 2D-to-3D keypoint matching;
determining a current location within the 3D map based on identifying the target image; and
autonomously or semi-autonomously navigating through the current environment based on determining the current location.
2. The method of claim 1 , wherein the second set of keypoints include features of the current environment.
3. The method of claim 1 , further comprising:
receiving the 3D map from a remote device; and
storing the 3D map at the vehicle.
4. The method of claim 3 , further comprising:
transmitting the query image to the remote device; and
receiving an updated 3D map based on the transmitted query image, the updated 3D map comprising updated keypoints based on the query image.
5. The method of claim 1 , wherein the first set of keypoints are generated based on a set of images.
6. The method of claim 5 , wherein each of the set of images is captured at a different time with a different sensor.
7. The method of claim 1 , wherein a distance between a location corresponding to the query image and a location corresponding to the target image satisfies a distance condition.
8. An apparatus for localization performed at an agent, comprising:
a processor;
a memory coupled with the processor; and
instructions stored in the memory and operable, when executed by the processor, to cause the apparatus to:
capture, via one or more sensors integrated with the vehicle, a query image of a current environment of the vehicle, the query image being a two-dimensional (2D) image;
identify a target image, from a plurality of images associated with a three-dimensional (3D) map of the current environment, comprising a first set of keypoints that match a second set of keypoints of the query image, each image of the plurality of images associated with a respective set of keypoints that are labelled via a keypoint model trained for 2D-to-3D keypoint matching;
determine a current location within the 3D map based on identifying the target image; and
autonomously or semi-autonomously navigate through the current environment based on determining the current location.
9. The apparatus of claim 8 , wherein the second set of keypoints include features of the current environment.
10. The apparatus of claim 8 , wherein execution of the instructions further cause the apparatus to:
receive the 3D map from a remote device; and
store the 3D map at the vehicle.
11. The apparatus of claim 10 , wherein execution of the instructions further cause the apparatus to:
transmit the query image to the remote device; and
receive an updated 3D map based on the transmitted query image, the updated 3D map comprising updated keypoints based on the query image.
12. The apparatus of claim 8 , wherein the first set of keypoints are generated based on a set of images.
13. The apparatus of claim 12 , wherein each of the set of images is captured at a different time with a different sensor.
14. The apparatus of claim 8 , wherein a distance between a location corresponding to the query image and a location corresponding to the target image satisfies a distance condition.
15. A non-transitory computer-readable medium having program code recorded thereon for localization performed at an agent, the program code executed by a processor and comprising:
program code to capture, via one or more sensors integrated with the vehicle, a query image of a current environment of the vehicle, the query image being a two-dimensional (2D) image;
program code to identify a target image, from a plurality of images associated with a three-dimensional (3D) map of the current environment, comprising a first set of keypoints that match a second set of keypoints of the query image, each image of the plurality of images associated with a respective set of keypoints that are labelled via a keypoint model trained for 2D-to-3D keypoint matching;
program code to determine a current location within the 3D map based on identifying the target image; and
program code to autonomously or semi-autonomously navigate through the current environment based on determining the current location.
16. The non-transitory computer-readable medium of claim 15 , wherein the second set of keypoints include features of the current environment.
17. The non-transitory computer-readable medium of claim 15 , wherein the program code further comprises:
program code to receive the 3D map from a remote device; and
program code to store the 3D map at the vehicle.