IP Library Granted Patent US 10,997,750
Granted Patent B1
US 10,997,750 · App. 16/731,902 · Granted May 4, 2021

Overhead view image generation

Inventors: Clemens Marschner (Munich, DE); Thomas Schiwietz (Munich, DE); Wilhelm Richert (Feldkirchen, DE); Nikolai Morin (Munich, DE); Holger Rapp (Munich, DE)
Assignee: Lyft, Inc.
G06T7/90G06T7/74G06T2207/10024G06T2207/20021
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,997,750
App. No.
16/731,902
Granted
May 4, 2021
Kind
B1
Abstract

The present invention relates to a method of generating an overhead view image of an area. More particularly, the present invention relates to a method of generating a contextual multi-image based overhead view image of an area using ground map data and field of view image data. Various embodiments of the present technology can include methods, systems and non-transitory computer readable media and computer programs configured to receive a plurality of images of the geographical area, determine a ground map of the geographical area, divide the ground map into a plurality of sampling points of the geographical area; and determine a color for each of the plurality of sampling points, wherein the color of each of the sampling points is determined by determining a correlation between the sampling points of the geographical area and color of the sampling points captured in at least one of the plurality of images.

Claims (47)

1. A method comprising;

receiving a plurality of images captured from at least one camera having ground level perspectives of a geographical area, the plurality of images captured over a plurality of trajectories and associated with respective fields of view;

determining a ground surface of the geographical area;

dividing the ground surface into a plurality of sampling points, the plurality of sampling points including at least a specified sampling point;

determining a subset of images including the specified sampling point, the subset of images including at least a first image and a second image of the plurality of images, wherein the first and second images are associated with different trajectories and different fields of view that include the specified sampling point;

determining a sampling color of the specified sampling point by identifying a color of at least one pixel of the subset of images that correlates to the specified sampling point; and

generating an overhead view image having an elevated perspective of the geographical area relative to the ground level perspectives based at least on the sampling color of the specified sampling point.

2. The method as recited in claim 1 , wherein determining the sampling color of the specified sampling point comprises:

receiving a plurality of colors of a plurality of pixels of the subset of images correlating to the specified sampling point; and

determining an average color of the plurality of colors to determine the sampling color of the specified sampling point.

3. The method as recited in claim 1 , wherein a fully transparent pixel is set for an undefined section of a ground map of the geographical area.

4. The method as recited in claim 1 , wherein determining the sampling color of the specified sampling point comprises:

determining a camera view ray for each image of the plurality of images in relation to the geographical area of each sampling point; and

wherein determining the sampling color of the specified sampling point is based on the color of the at least one pixel of the image at a region on the image at which an intersection of the camera view ray is determined to represent the specified sampling point of the geographic area based on the ground surface.

5. The method as recited in claim 4 , wherein determining the sampling color of the specified sampling point further comprises using data associated with the ground surface, the intersection of the camera view ray determined to represent the specified sampling point of the geographic area, and the plurality of images of the geographical area to determine which patches of the plurality of images relate to the specified sampling point of the ground surface.

6. The method as recited in claim 1 , wherein determining the sampling color of the specified sampling point comprises:

determining a three dimensional position of each sampling point to indicate whether a color is to be assigned to each sampling point, wherein each sampling point is queried against a geometric map to determine the three dimensional position for at least a region of the at least one image of the plurality of images.

7. The method as recited in claim 1 , wherein determining the sampling color of the specified sampling point comprises filtering the plurality of images based on a predetermined distance of the at least one camera to the geographical area of the specified sampling point.

8. The method as recited in claim 1 , wherein dividing the ground surface into the plurality of sampling points comprises dividing the ground surface into any one or any combination of: square grids, tiles, and/or hierarchical spatial data structures.

9. The method as recited in claim 1 , further comprising:

receiving pose data associated with each image of the plurality of images; and

determining, for each sampling point, a set of the plurality of images that capture at least a portion of a view of the sampling point based on the pose data associated with each image of the plurality of images.

10. The method as recited in claim 1 , wherein, prior to determining the ground surface of the geographical area, the method further comprises:

removing one or more images of the plurality of images by filtering the plurality of images.

11. The method as recited in claim 1 , further comprising determining a correlation between a three-dimensional coordinate of the ground surface and a two-dimensional coordinate in the at least one image of plurality of images that correspond to the three-dimensional coordinate.

12. The method as recited in claim 1 , wherein the sampling color of the specified sampling point is determined by incorporating the color of the at least one pixel without incorporating a color of a remaining pixel of at least one remaining image of the subset of images.

13. The method as recited in claim 1 , wherein data associated with the ground surface is generated using any one or any combination of: light detection and ranging (LiDAR), radio detection and ranging (RADAR), sound navigation and ranging (SONAR) or imaging cameras.

14. The method as recited in claim 1 , wherein the ground surface comprises an indication of elevation variances of the ground surface of the geographical area, the elevation variances providing a distinction between a drivable road surface and an undrivable road surface.

15. The method as recited in claim 1 , wherein the plurality of images are captured by two or more road vehicles each equipped with at least one image sensor.

16. The method as recited in claim 1 , wherein the plurality of images are captured at different times and/or different environmental conditions.

17. The method as recited in claim 1 , wherein the sampling points comprise a sampling area or a sampling volume.

18. The method as recited in claim 1 , wherein generating the overhead view image comprises aggregating the color of the at least one pixel of the subset of images that correlates to the specified sampling point of the ground surface.

19. A non-transitory computer-readable medium comprising computer-executable instructions which, when executed by at least one processor of a system, cause the system to perform:

receiving a plurality of images captured from at least one camera having ground level perspectives of a geographical area, the plurality of images captured over a plurality of trajectories and associated with respective fields of view;

determining a ground surface of the geographical area;

dividing the ground surface into a plurality of sampling points, the plurality of sampling points including at least a specified sampling point;

determining a subset of images including the specified sampling point, the subset of images including at least a first image and a second image of the plurality of images, wherein the first and second images are associated with different trajectories and different fields of view that include the specified sampling point;

determining a sampling color of the specified sampling point by identifying a color of the subset of images that correlates to the specified sampling point; and

generating an overhead view image having an elevated perspective of the geographical area relative to the ground level perspectives based at least on the sampling color of the specified sampling point.

20. A system comprising:

at least one processor and a memory storing instructions that, when executed by the at least one processor, cause the system to perform:

receiving a plurality of images captured from at least one camera having ground level perspectives of a geographical area, the plurality of images captured over a plurality of trajectories and associated with respective fields of view;

determining a ground surface of the geographical area;

dividing the ground surface into a plurality of sampling points, the plurality of sampling points including at least a specified sampling point;

determining a subset of images including the specified sampling point, the subset of images including at least a first image and a second image of the plurality of images, wherein the first and second images are associated with different trajectories and different fields of view that include the specified sampling point;

determining a sampling color of the specified sampling point by identifying a color of at least one pixel of the subset of images that correlates to the specified sampling point; and

generating an overhead view image having an elevated perspective of the geographical area relative to the ground level perspectives based at least on the sampling color of the specified sampling point.

Assignments (4)
CHANGE OF NAME Recorded Jun 22, 2023
From: WOVEN PLANET NORTH AMERICA, INC.
To: WOVEN BY TOYOTA, U.S., INC.
Reel/Frame 064065/0601 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 20, 2021
From: LYFT, INC.; BLUE VISION LABS UK LIMITED
To: WOVEN PLANET NORTH AMERICA, INC.
Reel/Frame 056927/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 12, 2021
From: LYFT, INC.; MAGNA AUTONOMOUS SYSTEMS, LLC
To: LYFT, INC.; MAGNA AUTONOMOUS SYSTEMS, LLC
Reel/Frame 057434/0623 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 27, 2020
From: MARSCHNER, CLEMENS; SCHIWIETZ, THOMAS; RICHERT, WILHELM; MORIN, NIKOLAI; RAPP, HOLGER
To: LYFT, INC.
Reel/Frame 051629/0649 →