IP Library Granted Patent US 9,164,511
Granted Patent B1
US 9,164,511 · App. 13/864,336 · Granted Oct 20, 2015

Use of detected objects for image processing

View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,164,511
App. No.
13/864,336
Granted
Oct 20, 2015
Kind
B1
Abstract

Methods and systems for the use of detected objects for image processing are described. A computing device autonomously controlling a vehicle may receive images of the environment surrounding the vehicle from an image-capture device coupled to the vehicle. In order to process the images, the computing device may receive information indicating characteristics of objects in the images from one or more sources coupled to the vehicle. Examples of sources may include RADAR, LIDAR, a map, sensors, a global positioning system (GPS), or other cameras. The computing device may use the information indicating characteristics of the objects to process received images, including determining the approximate locations of objects within the images. Further, while processing the image, the computing device may use information from sources to determine portions of the image to focus upon that may allow the computing device to determine a control strategy based on portions of the image.

Claims (46)

1. A method comprising:

receiving, at a computing device from an image-capture device coupled to a vehicle, an image depicting an object in an environment of the vehicle;

receiving, at the computing device via one or more sources coupled to the vehicle, information indicating a plurality of characteristics of the object that include a longitudinal speed and a lateral speed of the object relative to the vehicle;

based on the plurality of characteristics of the object, determining a type of the object;

identifying by the computing device, using the information indicating the plurality of characteristics of the object and a structure associated with the determined type of the object, a particular portion of a plurality of portions of the image that depicts a given feature of the object;

processing the particular portion of the image to determine a control strategy for the vehicle based on one or more parameters associated with the given feature of the object and the plurality of characteristics of the object; and

providing instructions to one or more systems of the vehicle to control the vehicle based on the determined control strategy for the vehicle.

2. The method of claim 1 , wherein the one or more sources associated with the vehicle include one or more of a laser, radio detection and ranging (RADAR), laser rangefinder and light detection and ranging (LIDAR), a map, sensors, or a global positioning system (GPS).

3. The method of claim 1 , wherein identifying the particular portion of the image comprises transforming a plurality of points of the object to a plurality of pixels within the image.

4. The method of claim 1 , wherein the information indicating the plurality of characteristics of the object further includes information indicating whether the object is moving or stationary, a direction of motion of the object, a size of the object, and a respective position of the object on a road on which the vehicle is traveling.

5. The method of claim 1 , wherein the computing device is configured to control the vehicle in an autonomous operation mode.

6. The method of claim 1 , wherein identifying the particular portion of the image comprises determining whether the object is at least partially occluded.

7. The method of claim 6 , wherein determining whether the object is at least partially occluded comprises:

determining one or more intersections in the particular portion of the image; and

determining if the object should be visible at the one or more intersections based on the information indicating one or more characteristics of the object.

8. The method of claim 1 , wherein receiving, at the computing device via one or more sources coupled to the vehicle, information indicating the plurality of characteristics of the object comprises receiving the one or more characteristics wirelessly from another computing device.

9. The method of claim 1 , further comprising:

determining, based on the particular portion of the image, one or more additional features of the object, wherein determining the one or more additional features of the object includes determining one or more of a brake signal, a turn signal, color of the object, or a license plate number.

10. A non-transitory computer readable medium having stored thereon executable instructions that, upon execution by a computing device, cause the computing device to perform functions comprising:

receiving, from an image-capture device coupled to a vehicle, an image depicting an object in an environment of the vehicle;

receiving, via one or more sources coupled to the vehicle, information indicating a plurality of characteristics of the object that include a longitudinal speed and a lateral speed of the object relative to the vehicle;

based on the plurality of characteristics of the object, determining a type of the object;

identifying, using the information indicating the plurality of characteristics of the object and a structure associated with the determined type of the object, a particular portion of a plurality of portions of the image that depicts a given feature of the object;

processing the particular portion of the image to determine a control strategy for the vehicle based on one or more parameters associated with the given feature of the object and the plurality of characteristics of the object; and

providing instructions to one or more systems of the vehicle to control the vehicle based on the determined control strategy for the vehicle.

11. The non-transitory computer readable medium of claim 10 , wherein the information indicating the plurality of characteristics of the object further includes one or more of information indicating whether the object is moving or stationary, a direction of motion of the object, a size of the object, and a respective position of the object on a road on which the vehicle is traveling.

12. The non-transitory computer readable medium of claim 10 , wherein the function of identifying the particular portion of the image comprises determining whether the object is at least partially occluded.

13. The non-transitory computer readable medium of claim 12 , wherein determining whether the object is at least partially occluded comprises:

determining one or more intersections in the particular portion of the image; and

determining if the object should be visible at the one or more intersections based on the information indicating one or more characteristics of the object.

14. The non-transitory computer readable medium of claim 10 , wherein the functions further comprise determining, based on the particular portion of the image, one or more additional features of the object, wherein determining the one or more additional features of the object includes determining one or more of a brake signal, a turn signal, color of the object, or license plate number.

15. A system comprising:

at least one processor; and

a memory having stored thereon instructions that, upon execution by the at least one processor, cause the control system to perform functions comprising:

receiving, from an image-capture device coupled to a vehicle, an image depicting an object in an environment of the vehicle;

receiving, via one or more sources coupled to the vehicle, information indicating a plurality of characteristics of the object that include a longitudinal speed and a lateral speed of the object relative to the vehicle;

based on the plurality of characteristics of the object, determining a type of the object;

identifying, using the information indicating the plurality of characteristics of the object and a structure associated with the determined type of the object, a particular portion of a plurality of portions of the image that depicts a given feature of the object;

processing the particular portion of the image to determine a control strategy for the vehicle based on one or more parameters associated with the given feature of the object and the plurality of characteristics of the object; and

providing instructions to one or more systems of the vehicle to control the vehicle based on the determined control strategy for the vehicle.

16. The system of claim 15 , wherein the function of identifying the particular portion of the image comprises determining at least one pixel within the image that matches to the information indicating the plurality characteristics of the object.

17. The system of claim 15 , wherein the one or more sources associated with the vehicle include one or more of a laser, radio detection and ranging (RADAR), laser rangefinder and light detection and ranging (LIDAR), a map, sensors, or a global positioning system (GPS).

18. The system of claim 15 , wherein the function of identifying the particular portion of the image comprises determining whether the object is at least partially occluded.

19. The system of claim 15 , wherein the function of identifying the particular portion of the image comprises transforming a plurality of points of the object to a plurality of pixels within the image.

20. The system of claim 15 , wherein the functions further comprise:

determining, based on the particular portion of the image, one or more additional features of the object, wherein determining the one or more additional features of the object includes determining one or more of a brake signal, a turn signal, color of the object, or license plate number.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 24, 2017
From: GOOGLE INC.
To: WAYMO HOLDING INC.
Reel/Frame 042084/0741 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 24, 2017
From: WAYMO HOLDING INC.
To: WAYMO LLC
Reel/Frame 042085/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 17, 2013
From: FERGUSON, DAVE; OGALE, ABHIJIT
To: GOOGLE INC.
Reel/Frame 030231/0031 →