IP Library › Granted Patent US 10,699,135
Granted Patent B2
US 10,699,135 · App. 15/818,230 · Granted Jun 30, 2020

Automatic localization geometry generator for stripe-shaped objects

Inventors: Qilin Zhang (Chicago, IL); Xiang Ma (Lisle, IL); Xin Chen (Evanston, IL); Sanjay Sood (Evanston, IL); Mark Tabb (Chicago, IL); Chen Luo (Chicago, IL)
Assignee: HERE Global B.V.
G06K9/00798G05D1/0212G06K9/4609G06K9/6256G06K9/6274G08G1/167G01C21/34G05D2201/0213
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,699,135
App. No.
15/818,230
Granted
Jun 30, 2020
Kind
B2
Abstract

Apparatus and methods are described for generating geometries for stripe-shaped objects. An image is identified that includes a roadway having one or more stripe-shaped objects. The stripe-shaped objects may include lane lines for road edges or lanes of the roadway. The stripe-shaped objects may include a barrier. At least one targeted region within the image is determined. The at least one targeted region is shaped to intersect the one or more stripe-shaped objects and includes a plurality of pixels. An image analysis is performed on the image to determine when the at least one target region includes a pixel in common with the one or more stripe-shaped objects. A geometry is constructed using the pixel in common. The geometry may be used to update a map or subsequently perform localization.

Claims (47)

1. A method for automatic localization geometry generation for stripe-shaped objects, the method comprising:

identifying an image including a roadway having one or more stripe-shaped objects;

determining, using a processor, at least one targeted region within the image, wherein the at least one targeted region is shaped to intersect the one or more stripe-shaped objects and overlaps a plurality of pixels in the image;

applying the image including the at least one targeted region to a multilayer neural network;

receiving, from the multilayer neural network, an indication whether the at least one targeted region having the plurality of pixels includes at least one pixel in common with the one or more stripe-shaped objects;

constructing a geometry of the one or more stripe-shaped objects using the at least one pixel in common; and

modifying the geometry of the one or more stripe-shaped objects by correlating the at least one pixel in common with a subset of data points of a three-dimensional point cloud.

2. The method of claim 1 , wherein the at least one targeted region includes a plurality of targeted regions spaced apart by a predetermined distance.

3. The method of claim 1 , wherein the at least one targeted region is U-shaped.

4. The method of claim 1 , wherein the one or more stripe-shaped objects include a lane marker, a road edge marker, a guardrail, a physical divider, or a combination thereof.

5. The method of claim 1 , wherein the one or more stripe-shaped objects are parallel to the roadway.

6. The method of claim 1 , wherein the multilayer neural network is trained based on ground truth locations of the one or more stripe-shaped objects in a plurality of sets of image data associated with the roadway.

7. The method of claim 1 , further comprising:

performing a driving assistance function in response to the image and analysis and whether the at least one target region includes the pixel in common with the one or more stripe-shaped objects.

8. The method of claim 1 , further comprising:

generating a driving command in response to the image and analysis and whether the at least one target region includes the pixel in common with the one or more stripe-shaped objects.

9. The method of claim 1 , wherein the at least one targeted region includes a plurality of targeted regions, the method further comprising:

identifying an intersection point with the one or more stripe-shaped for each of the plurality of targeted regions; and

calculating an average of the intersection points for the plurality of targeted regions, wherein the pixel in common corresponds to the average of the intersection points.

10. The method of claim 1 , wherein modifying the geometry of the one or more stripe-shaped objects by correlating the at least one pixel in common with the subset of data points of the three-dimensional point cloud further comprises:

identifying an area in the image corresponding to the subset of data points in the three-dimensional point cloud; and

performing a line connection algorithm using the area in the image and the subset of the data points in the three-dimensional point cloud.

11. The method of claim 1 , further comprising:

calculating a polyline based on the pixel in common with the one or more stripe-shaped object and a predetermined minimum deviation angle.

12. The method of claim 11 , wherein the polyline extends from the pixel in common with the one or more stripe-shaped object to a subsequent pixel located within the predetermined minimum deviation angle, the subsequent pixel received from the multilayer neural network and included in the plurality of pixels in the at least one targeted region.

13. An apparatus for automatic localization geometry generation for stripe-shaped objects, the apparatus comprising:

a database configured to store an image including a roadway having one or more stripe-shaped objects; and

a server configured to determine localization geometry from at least one targeted region within the image, wherein the at least one targeted region is shaped to intersect the one or more stripe-shaped objects and includes a plurality of pixels, the server configured to apply the image including the at least one targeted region to a multilayer neural network for image analysis, wherein the image analysis determines when the at least one targeted region includes at least one pixel in common with the one or more stripe-shaped objects, constructs a geometry of the one or more stripe-shaped objects using the at least one pixel in common, and modifies the geometry of the one or more stripe-shaped objects through a correlation of the at least one pixel in common with a subset of data points of a three-dimensional point cloud.

14. The apparatus of claim 13 , wherein the at least one targeted region includes a plurality of targeted regions spaced apart by a predetermined distance.

15. The apparatus of claim 13 , wherein the one or more stripe-shaped objects are parallel to the roadway and include a lane marker, a road edge marker, a guardrail, a physical divider, or a combination thereof.

16. The apparatus of claim 13 , further comprising:

a driving assistance device configured to perform a driving assistance function in response to the image and analysis and whether the at least one target region includes the pixel in common with the one or more stripe-shaped objects.

17. The apparatus of claim 16 , wherein the driving assistance device is configured to generate a driving command in response to the image and analysis and whether the at least one target region includes the pixel in common with the one or more stripe-shaped objects.

18. An apparatus comprising at least one processor and at least one memory including computer program code being executable by the at least one processor to cause the apparatus to perform operations comprising:

identifying an image including a roadway having one or more stripe-shaped objects;

determining, using a processor, at least one targeted region within the image, wherein the at least one targeted region is shaped to intersect the one or more stripe-shaped objects and overlaps a plurality of pixels in the image;

applying the image including the at least one targeted region to a multilayer neural network;

receiving, from the multilayer neural network, an indication whether the at least one targeted region having the plurality of pixels includes at least one pixel in common with the one or more stripe-shaped objects;

constructing a geometry of the one or more stripe-shaped objects using the at least one pixel in common; and

modifying the geometry of the one or more stripe-shaped objects by correlating the at least one pixel in common with a subset of data points of a three-dimensional point cloud.

19. A non-transitory computer-readable medium storing computer program code, the computer program code when executed by a processor causing an apparatus to perform and/or control:

identifying an image including a roadway having one or more stripe-shaped objects;

determining, using a processor, at least one targeted region within the image, wherein the at least one targeted region is shaped to intersect the one or more stripe-shaped objects and overlaps a plurality of pixels in the image;

applying the image including the at least one targeted region to a multilayer neural network;

receiving, from the multilayer neural network, an indication whether the at least one targeted region having the plurality of pixels includes at least one pixel in common with the one or more stripe-shaped objects;

constructing a geometry of the one or more stripe-shaped objects using the at least one pixel in common; and

modifying the geometry of the one or more stripe-shaped objects by correlating the at least one pixel in common with a subset of data points of a three-dimensional point cloud.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 6, 2019
From: TABB, MARK
To: HERE GLOBAL B.V.
Reel/Frame 049395/0360 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 18, 2018
From: ZHANG, QILIN; MA, XIANG; SOOD, SANJAY; LUO, CHEN; CHEN, XIN
To: HERE GLOBAL B.V.
Reel/Frame 047219/0859 →
Continuity (1)
Related Publication 20190156128A1 · May 23, 2019
Cited By (1)
US 12,272,131