Localization using semantically segmented images
Techniques are discussed for determining a location of a vehicle in an environment using a feature corresponding to a portion of an image representing an object in the environment which is associated with a frequently occurring object classification. For example, an image may be received and semantically segmented to associate pixels of the image with a label representing an object of an object type (e.g., extracting only those portions of the image which represent lane boundary markings). Features may then be extracted, or otherwise determined, which are limited to those portions of the image. In some examples, map data indicating a previously mapped location of a corresponding portion of the object may be used to determine a difference. The difference (or sum of differences for multiple observations) are then used to localize the vehicle with respect to the map.
1. A method comprising:
receiving a first image and a second image, the first image and the second image depicting a representation of an object in an environment;
inputting the first image and the second image into a machine-learned model trained to segment images;
receiving, from the machine-learned model, a first segmented image and a second segmented image, wherein a first portion of the first segmented image corresponds to the object depicted in the first image, and a second portion of the second segmented image corresponds to the object depicted in the second image, the object having a repeated object classification;
determining, in the first image, a first feature associated with the object;
determining, in the second image, a second feature associated with the object;
determining a first position of the first feature in the environment;
determining a second position of the second feature in the environment;
combining the first position and the second position to determine a combined position; and
storing the combined position as a landmark location.
2. The method of claim 1 , wherein combining the first position and the second position comprises performing a weighted average of the first position and the second position.
3. The method of claim 2 , wherein weights of the weighted average are based at least in part on one or more of:
a distance from an image sensor used to capture the first image to the first position;
a distance from the image sensor used to capture the second image to the second position;
an angle between an orientation of the image sensor used to capture the first image and the first position; or
an angle between an orientation of the image sensor used to capture the second image and the second position.
4. The method of claim 1 , wherein at least one of the first image or the second image is captured by a camera, and wherein determining the combined position comprises one or more of:
determining a point in the environment at which a ray from a camera center of the camera passing through the first feature or the second feature intersects a three-dimensional map; or
determining a plane proximate the camera; and
determining where the ray intersects the plane.
5. The method of claim 1 , wherein determining the combined position comprises performing a bundle adjustment based at least in part on the first position and the second position.
6. The method of claim 1 , wherein the first image or the second image are received from log data.
7. The method of claim 1 , wherein at least one of the first image or the second image is captured by a camera, and the at least one of the first image or the second image is associated with a pose of the camera at a first time at which the first image was taken or a second time at which the second image was taken.
8. A system comprising:
one or more processors; and
one or more computer-readable media storing instructions which, when executed by the one or more processors, cause the system to perform operations comprising:
receiving an image depicting an object in an environment;
inputting the image into a machine-learned model trained to segment images;
receiving, from the machine-learned model, a segmented image, wherein a portion of the segmented image corresponds to the object depicted in the image, the object having a repeated object classification;
determining a feature associated with the object;
determining a first position of the feature in the environment;
receiving a second position of an additional feature that is associated with the feature;
combining the first position and the second position to determine a combined position; and
storing the combined position as a landmark location.
9. The system of claim 8 , wherein combining the first position and the second position comprises performing a weighted average of the first position and the second position.
10. The system of claim 9 , wherein at least one weight of the weighted average is based at least in part on:
a distance from a camera used to capture the image to the first position; or
an angle between an orientation of the camera used to capture the image and the first position.
11. The system of claim 8 , wherein determining the combined position comprises at least one of:
determining a point in the environment at which a ray from a camera center of a camera used to capture the image and passing through the feature intersects a three-dimensional map; or
determining a plane proximate the camera; and
determining where the ray intersects the plane.
12. The system of claim 8 , wherein determining the combined position comprises performing a bundle adjustment based at least in part on the first position and the second position.
13. The system of claim 8 , wherein the image is a first image, the segmented image is a first segmented image, and the feature is a first feature, the operations further comprising:
receiving, from log data, a second image depicting the object;
inputting the second image into the machine-learned model;
receiving, from the machine-learned model, a second segmented image, wherein a portion of the second segmented image corresponds to the object depicted in the second image; and
determining the additional feature associated with the object in the second image,
wherein receiving the second position is based at least in part on determining a second position associated with the additional feature in the environment.
14. The system of claim 8 , wherein the image is associated with a pose of a camera used to capture the image at a time at which the image was taken.
15. One or more computer-readable media storing instructions, which when executed by one or more processors, perform operations comprising:
receiving an image depicting an object in an environment;
inputting the image into a machine-learned model trained to segment images;
receiving, from the machine-learned model, a segmented image, wherein a portion of the segmented image corresponds to the object depicted in the image, the object having a repeated object classification;
determining a feature associated with the object;
determining a first position of the feature in the environment;
receiving a second position of an additional feature in the environment associated with an additional image;
combining the first position and the second position to determine a combined position; and
storing the combined position as a landmark location.
16. The one or more computer-readable media of claim 15 , wherein the image is received from a camera of a vehicle traversing the environment.
17. The one or more computer-readable media of claim 15 , wherein the landmark location is stored as part of a map of the environment.
18. The one or more computer-readable media of claim 17 , the operations further comprising outputting the map including the landmark location to a vehicle.
19. The one or more computer-readable media of claim 15 , wherein the second position is received from log data.
20. The one or more computer-readable media of claim 15 , wherein combining the first position and the second position comprises performing a weighted average of the first position and the second position.