Method and system for video-based positioning and mapping
A method and system for determining a geographical location and orientation of a vehicle travelling through a road network is disclosed. The method comprises obtaining, from one or more cameras associated with the vehicle travelling through the road network, a sequence of images reflecting the environment of the road network on which the vehicle is travelling, wherein each of the images has an associated camera location at which the image was recorded. A local map representation representing an area of the road network on which the vehicle is travelling is then generated using at least some of the obtained images and the associated camera locations. The generated local map representation is compared with a section of a reference map, the reference map section covering the area of the road network on which the vehicle is travelling, and the geographical location and orientation of the vehicle within the road network is determined based on the comparison. Methods and systems for generating and/or updating an electronic map using data obtained by a vehicle travelling through a road network represented by the map are also disclosed.
1. A method, comprising:
obtaining, from at least one camera associated with a vehicle traveling on a road, a sequence of images of an environment of the road, each image being associated with a location where that image was obtained;
generating a local map representation of an area of the road using at least some images from the sequence of images and the locations associated therewith, the generating including:
processing the at least some of the images to detect a landmark object representing a landmark in the environment of the road, the processing including, for each image of the at least some of the images:
allocating at least one object class associated with landmark objects to a region in that image that includes the landmark object based on a result of a segmentation performed using a specified machine learning algorithm;
identifying the region as a region of interest based on the at least one object class allocated thereto; and
processing the region of interest to generate a bounding area in which the landmark object is included;
determining at least one transformation for mapping the landmark object between the at least some of the images; and
based on the at least one transformation and the locations associated with the at least some of the images, generating a three-dimensional representation for the landmark object relative to the area of the road;
comparing the local map representation with some or all of a reference map to identify a corresponding section of the reference map; and
selectively updating the corresponding section of the reference map based on the local map representation.
2. The method of claim 1 , wherein comparing the local map representation with some or all of the reference map includes:
comparing the three-dimensional representation for the landmark object to at least one reference landmark shape in the reference map.
3. The method of claim 1 , wherein selectively updating the corresponding section of the reference map using the local map representation includes:
determining, based on the local map representation, whether the corresponding section of the reference map is missing information about the landmark object and/or includes erroneous information for the landmark object; and
when the reference map is missing information or includes erroneous information, updating the corresponding section of the reference map based on the local map representation; and
when the reference map is not missing information and does not include erroneous information, leaving the reference map unchanged.
4. The method of claim 1 , wherein selectively updating the corresponding section of the reference map based on the local map representation includes:
adding, to the reference map, landmark observation information that describes at least one characteristic of the landmark object.
5. The method of claim 4 , wherein the at least one characteristic of the landmark object includes at least one of:
a location of the landmark object;
an orientation of the landmark object;
a two-dimensional (2D) polyline representing a shape of the landmark object;
a pose matrix for transforming the 2D polyline into a three dimensional coordinate space; and
a reference image describing content contained in the 2D polyline.
6. The method of claim 1 , wherein the landmark object represents a landmark in the environment of the road network, the landmark being one of a building, a traffic sign, a traffic light, or a billboard.
7. The method of claim 1 , wherein determining the at least one transformation for mapping the landmark object includes:
determining a change in position and/or rotation for the landmark object between sequential images based on a respective location of the at least one camera where each of the images was captured.
8. The method of claim 1 , wherein generating the local map representation includes:
adding, to the local map representation, landmark observation information including a description of at least one characteristic of the landmark object.
9. A device, comprising:
at least one processor configured to:
obtain, from at least one camera associated with a vehicle traveling on a road, a sequence of images of an environment of the road, each image being associated with a location where that image was obtained;
generate a local map representation of an area of the road using at least some images from the sequence of images and the locations associated therewith, the generating including:
processing the at least some of the images to detect a landmark object representing a landmark in the environment of the road, the processing including, for each image of the at least some of the images:
allocating at least one object class associated with landmark objects to a region in that image that includes the landmark object based on a result of a segmentation performed using a specified machine learning algorithm;
identifying the region as a region of interest based on the at least one object class allocated thereto; and
processing the region of interest to generate a bounding area in which the landmark object is included;
determining at least one transformation for mapping the landmark object between the at least some of the images; and
based on the at least one transformation and the locations associated with the at least some of the images, generating a three-dimensional representation for the landmark object relative to the area of the road;
compare the local map representation with some or all of a reference map to identify a corresponding section of the reference map; and
selectively update the corresponding section of the reference map based on the local map representation.
10. The device of claim 9 , wherein, when comparing the local map representation with some or all of the reference map, the processor is configured to:
compare the three-dimensional representation for the landmark object to at least one reference landmark shape in the reference map.
11. The device of claim 9 , wherein, when selectively updating the corresponding section of the reference map using the local map representation, the processor is configured to:
determine, based on the local map representation, whether the corresponding section of the reference map is missing information about the landmark object and/or includes erroneous information for the landmark object; and
when the reference map is missing information or includes erroneous information, update the corresponding section of the reference map based on the local map representation; and
when the reference map is not missing information and does not include erroneous information, leave the reference map unchanged.
12. The device of claim 9 , wherein, when selectively updating the corresponding section of the reference map based on the local map representation, the processor is configured to:
add, to the reference map, landmark observation information that describes at least one characteristic of the landmark object.
13. The device of claim 12 , wherein the at least one characteristic of the landmark object includes at least one of:
a location of the landmark object;
an orientation of the landmark object;
a two-dimensional (2D) polyline representing a shape of the landmark object;
a pose matrix for transforming the 2D polyline into a three dimensional coordinate space;
a reference image describing content contained in the 2D polyline.
14. The device of claim 9 , wherein, when determining the at least one transformation for mapping the landmark object, the processor is configured to:
determine a change in position and/or rotation for the landmark object between sequential images based on a respective location of the at least one camera where each of the images was captured.
15. The device of claim 9 , wherein, when generating the local map representation, the processor is configured to:
add, to the local map representation, landmark observation information including a description of at least one characteristic of the landmark object.
16. A non-transitory computer-readable storage medium storing instructions which, when executed by a processor in a device, cause the processor to perform a method, the method comprising:
obtaining, from at least one camera associated with a vehicle traveling on a road, a sequence of images of an environment of the road, each image being associated with a location where that image was obtained;
generating a local map representation of an area of the road using at least some images from the sequence of images and the locations associated therewith, the generating including:
processing the at least some of the images to detect a landmark object representing a landmark in the environment of the road, the processing including, for each image of the at least some of the images:
allocating at least one object class associated with landmark objects to a region in that image that includes the landmark object based on a result of a segmentation performed using a specified machine learning algorithm;
identifying the region as a region of interest based on the at least one object class allocated thereto; and
processing the region of interest to generate a bounding area in which the landmark object is included;
determining at least one transformation for mapping the landmark object between the at least some of the images; and
based on the at least one transformation and the locations associated with the at least some of the images, generating a three-dimensional representation for the landmark object relative to the area of the road;
comparing the local map representation with some or all of a reference map to identify a corresponding section of the reference map; and
selectively updating the corresponding section of the reference map based on the local map representation.