IP Library Granted Patent US 10,823,562
Granted Patent B1
US 10,823,562 · App. 16/244,925 · Granted Nov 3, 2020

Systems and methods for enhanced base map generation

Inventors: Jeremy Carnahan (Normal, IL); Michael Stine McGraw (Normal, IL); John Andrew Schirano (Bloomington, IL)
Assignee: STATE FARM MUTUAL AUTOMOBILE INSURANCE COMPANY
G01C11/06G06F16/29G06K9/00798G06T7/55G06T7/70G06T17/05G06T2207/10028G06T2207/30256
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,823,562
App. No.
16/244,925
Granted
Nov 3, 2020
Kind
B1
Abstract

A feature mapping computer system configured to (i) receive a first localized image including a first photo and a first location; (ii) receive a second localized image including a second photo and a second location; (iii) identify a roadway feature depicted in both the first and second photos; (iv) generate, using a photogrammetry module, a point cloud based upon the first and second photos and first and second locations; (v) generate a localized point cloud by assigning a location to the point cloud based upon at least one of the first and second locations; and (vi) generate an enhanced base map that includes a roadway feature by embedding an indication of the identified roadway feature onto the localized point cloud.

Claims (86)

1. A feature mapping computer system for generating an enhanced base map, the feature mapping computer system including at least one processor in communication with at least one memory device, wherein the at least one processor is configured to:

receive a first localized image including a first photo depicting a driving environment, a first camera angle describing the camera orientation at which the first photo was captured, and a first location associated with the first photo;

receive a second localized image including a second photo depicting the driving environment, a second camera angle describing the camera orientation at which the second photo was captured, and a second location associated with the second photo, wherein the second location is different from the first location;

identify, using an image recognition module, a roadway feature depicted in both the first and second photos;

generate, using a photogrammetry module, a point cloud based upon the first and second photos, the first and second camera angles, and the first and second locations, wherein the point cloud comprises a set of data points representing the driving environment in a three dimensional (“3D”) space;

generate a localized point cloud by assigning a location to the point cloud based upon at least one of the first and second locations; and

generate an enhanced base map that includes a roadway feature by embedding an indication of the identified roadway feature onto the localized point cloud.

2. The feature mapping computer system of claim 1 , wherein the processor is further configured to:

determine a feature attribute associated with the identified roadway feature; and

embed the feature attribute onto the localized point cloud.

3. The feature mapping computer system of claim 2 , wherein the feature attribute includes a color associated with the identified roadway feature.

4. The feature mapping computer system of claim 2 , wherein the feature attribute includes letters or numbers associated with the roadway feature.

5. The feature mapping computer system of claim 2 , wherein the identified roadway feature is a street sign and the feature attribute includes at least one of a color and a word associated with the street sign.

6. The feature mapping computer system of claim 1 , wherein the processor is further configured to:

determine a feature attribute associated with the identified roadway feature;

generate a classified roadway feature by associating the feature attribute with the identified roadway feature; and

embed an indication of the classified roadway feature onto the localized point cloud.

7. The feature mapping computer system of claim 1 , wherein the processor is further configured to:

identify, using the image recognition module, a second roadway feature depicted in both the first and second photos;

determine a feature attribute associated with the second identified roadway feature; and

determine that the second roadway feature should be omitted from the base map.

8. The feature mapping computer system of claim 1 , wherein the first and second photos are aerial images captured by a drone.

9. The feature mapping computer system of claim 1 , wherein the first and second photos are high-definition (“HD”) photos.

10. The feature mapping computer system of claim 1 , wherein the processor is further configured to:

receive a location correction factor from a database; and

determine a corrected location by applying the location correction factor to at least one of the first and second locations.

11. The feature mapping computer system of claim 1 , wherein the processor is further configured to:

determine a plurality of data points in the point cloud that represent the identified roadway feature;

determine a priority for each of the data points based upon the relevance of each point in accurately depicting the roadway feature; and

remove, from the point cloud, the data points which are determined to be below a certain priority threshold.

12. The feature mapping computer system of claim 1 , wherein the processor is further configured to:

identify, using the image recognition module, a roadway feature represented in the point cloud.

13. The feature mapping computer system of claim 1 , wherein the processor is further configured to transmit the enhanced base map to an autonomous vehicle system, wherein the enhanced base map is configured to be used by the autonomous system for localizing an autonomous vehicle.

14. The feature mapping computer system of claim 13 , wherein the data points of the enhanced base map are configured to allow comparison with data points generated by an autonomous vehicle system.

15. A computer implemented method for generating an enhanced base map, the method implemented by a computer system including at least one processor, the method comprising:

receiving a first localized image including a first photo depicting a driving environment, a first camera angle describing the camera orientation at which the first photo was captured, and a first location associated with the first photo;

receiving a second localized image including a second photo depicting the driving environment, a second camera angle describing the camera orientation at which the second photo was captured, and a second location associated with the second photo, wherein the second location is different from the first location;

identifying, using an image recognition module, a roadway feature depicted in both the first and second photos;

generating, using a photogrammetry module, a point cloud based upon the first and second photos, the first and second camera angles, and the first and second locations, wherein the point cloud comprises a set of data points representing the driving environment in a three dimensional (“3D”) space;

generating a localized point cloud by assigning a location to the point cloud based upon at least one of the first and second locations; and

generating an enhanced base map that includes a roadway feature by embedding an indication of the identified roadway feature onto the localized point cloud.

16. The computer implemented method of claim 15 , wherein the method further comprises:

determining a feature attribute associated with the identified roadway feature; and

embedding the feature attribute onto the localized point cloud.

17. The computer implemented method of claim 15 , wherein the method further comprises:

determining a feature attribute associated with the identified roadway feature;

generating a classified roadway feature by associating the feature attribute with the identified roadway feature; and

embedding an indication of the classified roadway feature onto the localized point cloud.

18. The computer implemented method of claim 15 , wherein the method further comprises:

identifying, using the image recognition module, a second roadway feature depicted in both the first and second photos;

determining a feature attribute associated with the second identified roadway feature; and

determining that the second roadway feature should be omitted from the base map.

19. The computer implemented method of claim 15 , wherein the method further comprises:

determining a plurality of data points in the point cloud that represent the identified roadway feature;

determining a priority for each of the data points based upon the relevance of each point in accurately depicting the roadway feature; and

removing, from the point cloud, the data points which are determined to be below a certain priority threshold.

20. The computer implemented method of claim 15 , wherein the method further comprises:

identifying, using the image recognition module, a roadway feature represented in the point cloud.

21. The computer implemented method of claim 15 , wherein the method further comprises transmitting the enhanced base map to an autonomous vehicle system, wherein the enhanced base map is configured to be used by the autonomous vehicle system for localizing an autonomous vehicle.

22. The computer implemented method of claim 21 , wherein the data points of the enhanced base map are configured to allow comparison with data points generated by an autonomous vehicle system.

23. At least one non-transitory computer-readable storage media having computer-executable instructions embodied thereon for generating an enhanced base map, wherein when executed by at least one processor, the computer-executable instructions cause the processor to:

receive a first localized image including a first photo depicting a driving environment, a first camera angle describing the camera orientation at which the first photo was captured, and a first location associated with the first photo;

receive a second localized image including a second photo depicting the driving environment, a second camera angle describing the camera orientation at which the second photo was captured, and a second location associated with the second photo, wherein the second location is different from the first location;

identify, using an image recognition module, a roadway feature depicted in both the first and second photos;

generate, using a photogrammetry module, a point cloud based upon the first and second photos, the first and second camera angles, and the first and second locations, wherein the point cloud comprises a set of data points representing the driving environment in a three dimensional (“3D”) space;

generate a localized point cloud by assigning a location to the point cloud based upon at least one of the first and second locations; and

generate an enhanced base map that includes a roadway feature by embedding an indication of the identified roadway feature onto the localized point cloud.

24. The computer-readable storage media of claim 23 , wherein the computer-executable instructions further cause the processor to:

determine a feature attribute associated with the identified roadway feature; and

embed the feature attribute onto the localized point cloud.

25. The computer-readable storage media of claim 23 , wherein the computer-executable instructions further cause the processor to:

determine a feature attribute associated with the identified roadway feature;

generate a classified roadway feature by associating the feature attribute with the identified roadway feature; and

embed an indication of the classified roadway feature onto the localized point cloud.

26. The computer-readable storage media of claim 23 , wherein the computer-executable instructions further cause the processor to:

identify, using the image recognition module, a second roadway feature depicted in both the first and second photos;

determine a feature attribute associated with the second identified roadway feature; and

determine that the second roadway feature should be omitted from the base map.

27. The computer-readable storage media of claim 23 , wherein the computer-executable instructions further cause the processor to:

determine a plurality of data points in the point cloud that represent the identified roadway feature;

determine a priority for each of the data points based upon the relevance of each point in accurately depicting the roadway feature; and

remove, from the point cloud, the data points which are determined to be below a certain priority threshold.

28. The computer-readable storage media of claim 23 , wherein the computer-executable instructions further cause the processor to:

identify, using the image recognition module, a roadway feature represented in the point cloud.

29. The computer-readable storage media of claim 23 , wherein the computer-executable instructions further cause the processor to transmit the enhanced base map to an autonomous vehicle system, wherein the enhanced base map is configured to be used by the autonomous system for localizing an autonomous vehicle.

30. The computer-readable storage media of claim 23 , wherein the data points of the enhanced base map are configured to allow comparison with data points generated by an autonomous vehicle system.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 10, 2019
From: CARNAHAN, JEREMY; MCGRAW, MICHAEL STINE; SCHIRANO, JOHN ANDREW
To: STATE FARM MUTUAL AUTOMOBILE INSURANCE COMPANY
Reel/Frame 047959/0289 →
Cited By (6)
US 12,203,772 US 12,245,050 US 12,387,374 US 12,394,217 US 12,462,485 US 12,483,801