Curb-based feature extraction for localization and lane detection using radar
Provided are methods, systems, and computer program products for lane keep tracking and/or localization of a vehicle using a radar to detect metallic particles in paint applied to lane markings or curb markings. An example method may include: causing a radar system of a vehicle to output radio waves; receiving a radar image from the radar system corresponding to returned radio waves; determining a portion of the radar image includes lane or curb markings, wherein the lane or curb markings are embedded with metallic particles; and determining a location of the vehicle based at least in part on the determined portion of the radar image that includes the lane or curb markings.
1 . A method, comprising:
causing a radio detection and ranging (radar) system of a vehicle to output radio waves;
receiving a first radar image from the radar system corresponding to radio waves detected by the radar system in response to the outputted radio waves;
identifying a portion of the first radar image that includes a curb and curb markings corresponding to the curb, based on metallic particles embedded within paint on the curb, wherein identifying the portion of the first radar image that includes the curb and the curb markings comprises:
generating a 3D point cloud from the first radar image based on a plurality of metallic particles within a portion of the paint,
determining a first 3D shape based on the 3D point cloud,
extracting a curb feature from the first 3D shape, wherein the curb feature includes a second 3D shape of at least one curb marking of the curb markings,
generating a signature based on the first 3D shape and the curb feature,
comparing the generated signature with a plurality of signatures, wherein comparing the generated signature with the plurality of signatures comprises comparing the first 3D shape with a plurality of 3D shapes and the curb feature with a plurality of curb features, and
determining, based on the comparing, that the generated signature matches at least one of the plurality of signatures, wherein the at least one of the plurality of signatures corresponds to a second radar image of the curb markings generated based on the metallic particles embedded within the paint on the curb; and
determining a location of the vehicle based on the identified portion of the first radar image that includes the curb markings.
2 . The method of claim 1 , wherein the portion of the first radar image that includes the curb markings comprises a set of radar points of the first radar image.
3 . The method of claim 1 , wherein the location of the vehicle is a relative location of the vehicle with respect to a curb associated with the curb markings.
4 . The method of claim 1 , wherein the location of the vehicle is a relative location of the vehicle with respect to an origin of a map.
5 . The method of claim 1 , wherein determining the location of the vehicle comprises determining a first location and a second location based on the identified portion of the first radar image that includes the curb markings, wherein the first location is a relative location of the vehicle with respect to a curb associated with the curb markings, and wherein the second location is a relative location of the vehicle with respect to an origin of a map.
6 . The method of claim 1 , wherein determining the location of the vehicle based on the identified portion of the first radar image that includes the curb markings comprises:
determining a state of the vehicle with respect to a curb associated with the curb markings based on the portion of the first radar image that includes the curb markings; and
causing the vehicle to move relative to the curb based on the determined state of the vehicle.
7 . The method of claim 6 , wherein determining the state of the vehicle with respect to the curb comprises:
determining ranges and azimuth angles of corresponding curb markings based on the portion of the first radar image that includes the curb markings, and
determining the state of the vehicle based on the determined ranges and azimuth angles of the corresponding curb markings, wherein the state of the vehicle comprises at least one of: a location of the vehicle in a lane, an orientation of the vehicle to the lane, a rate of change of the location of the vehicle in the lane, or a rate of change of the orientation of the vehicle to the lane.
8 . The method of claim 1 , wherein determining the location of the vehicle based on the identified portion of the first radar image that includes the curb markings comprises:
determining the location of the vehicle based on the curb feature.
9 . The method of claim 8 , wherein determining the location of the vehicle based on the curb feature comprises:
obtaining a digital map,
determining a matching feature of the digital map to the curb feature,
obtaining a match location for the matching feature,
determining a transformation from the match location based on at least the first radar image, and
determining the location of the vehicle based on the match location and the transformation.
10 . The method of claim 9 , further comprising:
updating a previous or estimated location of the vehicle using a Kalman filter and the determined location of the vehicle.
11 . The method of claim 9 , wherein the digital map includes at least light detection and ranging (lidar) point cloud data for curbs in a geographic area.
12 . The method of claim 1 , wherein extracting the curb feature from the first 3D shape comprises:
communicating the portion of the first radar image to a trained machine learning system; and
receiving the curb feature from the trained machine learning system.
13 . The method of claim 12 , wherein the trained machine learning system comprises a trained neural network trained to extract curb features from radar images.
14 . A system, comprising:
at least one processor, and
at least one non-transitory storage media storing instructions that, when executed by the at least one processor, cause the at least one processor to:
cause a radio detection and ranging (radar) system of a vehicle to output radio waves;
receive a first radar image from the radar system corresponding to radio waves detected by the radar system in response to the outputted radio waves;
identify a portion of the first radar image that includes a curb and curb markings corresponding to the curb, based on metallic particles embedded within paint on the curb, wherein to identify the portion of the first radar image that includes the curb and the curb markings, the instructions further cause the at least one processor to:
generate a 3D point cloud from the first radar image based on a plurality of metallic particles within a portion of the paint,
determine a first 3D shape based on the 3D point cloud,
extract a curb feature from the first 3D shape, wherein the curb feature includes a second 3D shape of at least one curb marking of the curb markings,
generate a signature based on the first 3D shape and the curb feature,
compare the generated signature with a plurality of signatures, wherein to compare the generated signature with the plurality of signatures, the instructions, when executed by the at least one processor, further cause the at least one processor to compare the first 3D shape with a plurality of 3D shapes and the curb feature with a plurality of curb features, and
determine, based on the comparison of the generated signature with the plurality of signatures, that the generated signature matches at least one of the plurality of signatures, wherein the at least one of the plurality of signatures corresponds to a second radar image of the curb markings generated based on the metallic particles embedded within the paint on the curb; and
determine a location of the vehicle based on the identified portion of the first radar image that includes the curb markings.
15 . The system of claim 14 , wherein the portion of the first radar image that includes the curb markings comprises a set of radar points of the first radar image.
16 . The system of claim 14 , wherein to determine the location of the vehicle based on the identified portion of the first radar image that includes the curb markings, the instructions further cause the at least one processor to:
determine a state of the vehicle with respect to a curb associated with the curb markings based on the portion of the first radar image that includes the curb markings; and
cause the vehicle to move relative to the curb based on the determined state of the vehicle.
17 . The system of claim 14 , wherein to determine the location of the vehicle based on the identified portion of the first radar image that includes the curb markings, the instructions further cause the at least one processor to:
determine the location of the vehicle based on the curb feature.
18 . The system of claim 17 , wherein to determine the location of the vehicle based on the curb feature, the instructions further cause the at least one processor to:
obtain a digital map,
determine a matching feature of the digital map to the curb feature,
obtain a match location for the matching feature,
determine a transformation from the match location based on at least the first radar image, and
determine the location of the vehicle based on the match location and the transformation.
19 . At least one non-transitory storage media storing instructions that, when executed by at least one processor, cause the at least one processor to:
cause a radio detection and ranging (radar) system of a vehicle to output radio waves;
receive a first radar image from the radar system corresponding to radio waves detected by the radar system in response to the outputted radio waves;
identify a portion of the first radar image that includes a curb and curb markings corresponding to the curb, based on metallic particles embedded within paint on the curb, wherein to identify the portion of the first radar image that includes the curb and the curb markings, the instructions further cause the at least one processor to:
generate a 3D point cloud from the first radar image based on a plurality of metallic particles within a portion of paint,
determine a first 3D shape based on the 3D point cloud,
extract a curb feature from the first 3D shape, wherein the curb feature includes a second 3D shape of at least one curb marking of the curb markings,
generate a signature based on the first 3D shape and the curb feature,
compare the generated signature with a plurality of signatures, wherein to compare the generated signature with the plurality of signatures, the instructions, when executed by the at least one processor, further cause the at least one processor to compare the first 3D shape with a plurality of 3D shapes and the curb feature with a plurality of curb features, and
determine, based on the comparison of the generated signature with the plurality of signatures, that the generated signature matches at least one of the plurality of signatures, wherein the at least one of the plurality of signatures corresponds to a second radar image of the curb markings generated based on the metallic particles embedded within the paint on the curb; and
determine a location of the vehicle based on the identified portion of the first radar image that includes the curb markings.
20 . The at least one non-transitory storage media of claim 19 , wherein the portion of the first radar image that includes the curb markings comprises a set of radar points of the first radar image.