IP Library Granted Patent US 9,607,226
Granted Patent B2
US 9,607,226 · App. 14/791,725 · Granted Mar 28, 2017

Methods and systems for object detection using multiple sensors

View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,607,226
App. No.
14/791,725
Granted
Mar 28, 2017
Kind
B2
Abstract

Methods and systems for object detection using multiple sensors are described herein. In an example embodiment, a vehicle's computing device may receive sensor data frames indicative of an environment at different rates from multiple sensors. Based on a first frame from a first sensor indicative of the environment at a first time period and a portion of a first frame that corresponds to the first time period from a second sensor, the computing device may estimate parameters of objects in the vehicle's environment. The computing device may modify the parameters in response to receiving subsequent frames or subsequent portions of frame of sensor data from the sensors even if the frames arrive at the computing device out of order. The computing device may provide the parameters of the objects to systems of the vehicle for object detection and obstacle avoidance.

Claims (49)

1. A method comprising:

receiving, at a computing system from a first sensor at a first rate, first sensor data corresponding to an environment of a vehicle;

receiving, at the computing system from a second sensor at a second rate, second sensor data corresponding to the environment;

determining, using probabilistic gating on a first data segment of the first sensor data and a portion of a first data segment of the second sensor data that correspond to the environment at a first time period, information indicative of parameters of one or more objects in the environment, wherein the probabilistic gating associates measurements within a given data segment of sensor data to respective objects in the environment;

modifying, using probabilistic gating on a second data segment of the first sensor data and a subsequent portion of the first data segment of the second sensor data that correspond to the environment at a second time period, the information indicative of the parameters of the one or more objects; and

providing, by the computing system, the modified information indicative of the parameters of the one or more objects to one or more systems of the vehicle.

2. The method of claim 1 , wherein the first sensor is LIDAR and the second sensor is a camera system.

3. The method of claim 1 , wherein the computing system completes receiving the first data segment of the first sensor data prior to completely receiving the first data segment of the second sensor data.

4. The method of claim 1 , wherein providing the modified information indicative of the parameters of the one or more objects to one or more systems of the vehicle includes providing obstacle avoidance instructions.

5. The method of claim 1 , wherein the second rate is less than the first rate.

6. The method of claim 1 , wherein determining information indicative of the parameters of the one or more objects in the environment comprises:

estimating a position and size of respective objects of the one or more objects in the environment.

7. The method of claim 1 , wherein modifying the information indicative of the parameters of the one or more objects comprises:

estimating a change in a velocity and an acceleration of respective objects of the one or more objects in the environment.

8. The method of claim 1 , wherein determining information indicative of the parameters of the one or more objects in the environment comprises:

estimating one or more trajectories of respective objects of the one or more objects, wherein the one or more trajectories are indicative of position patterns of the respective objects based on a near-term time period.

9. The method of claim 8 , wherein modifying the information indicative of the parameters of the one or more objects comprises:

adjusting the one or more trajectories of respective objects based on one or more changes in the information corresponding to the one or more objects.

10. The method of claim 1 , further comprising:

based on the modified information indicative of the parameters of the one or more objects, providing one or more conceptual gates that substantially outline respective objects of the one or more objects in the environment to the one or more systems of the vehicle.

11. A system comprising:

at least one processor; and

a memory having stored thereon instructions that, upon execution by the at least one processor, cause the system to perform functions comprising:

receiving, from a first sensor at a first rate, first sensor data corresponding to an environment of a vehicle;

receiving, from a second sensor at a second rate, second sensor data corresponding to the environment;

determining, using probabilistic gating on a first data segment of the first sensor data and a portion of a first data segment of the second sensor data that correspond to the environment at a first time period, information indicative of parameters of one or more objects in the environment, wherein the probabilistic gating associates measurements within a given data segment of sensor data to respective objects in the environment;

modifying, using probabilistic gating on a second data segment of the first sensor data and a subsequent portion of the first data segment of the second sensor data that correspond to the environment at a second time period, the information indicative of the parameters of the one or more objects; and

providing the modified information indicative of the parameters of the one or more objects to one or more systems of the vehicle.

12. The system of claim 11 , wherein receiving, from the first sensor at the first rate, first sensor data corresponding to the environment of a vehicle comprises:

receiving respective data segments of the first sensor data from the first sensor at a first latency level; and

wherein receiving, from the second sensor at the second rate, sensor data corresponding to the environment comprises:

receiving respective data segments of second sensor data from the second sensor at a second latency level, wherein the second latency level is a threshold higher than the first latency level.

13. The system of claim 11 , wherein determining, using the first data segment of the first sensor data and the portion of the first data segment of the second sensor data that correspond to the environment at the first time period, information indicative of parameters of one or more objects in the environment comprises:

estimating one or more trajectories of respective objects in the one or more objects responsive to receiving the first data segment of the first sensor data from the first sensor, wherein the one or more trajectories are indicative of near-term position patterns of the respective objects; and

adjusting the one or more trajectories of the respective objects in the one or more objects responsive to receiving the portion of the first data segment of the second sensor data from the second sensor.

14. The system of claim 13 , wherein providing the modified information indicative of the parameters of the one or more objects to one or more systems of the vehicle includes providing the one or more adjusted trajectories of the respective objects in the one or more objects to the one or more systems of the vehicle.

15. The system of claim 11 , wherein determining, using the first data segment of the first sensor data and the portion of the first data segment of the second sensor data that correspond to the environment at the first time period, information indicative of the parameters of one or more objects in the environment comprises:

using a Global Nearest Neighbor (GNN) process to select the portion of the first data segment of the second sensor data that corresponds to the environment at a same time of the first data segment of the first sensor data.

16. A non-transitory computer readable medium having stored therein instructions, that when executed by a computing device, cause the computing device to perform functions comprising:

receiving, from a first sensor at a first rate, first sensor data corresponding to an environment of a vehicle;

receiving, from a second sensor at a second rate, second sensor data corresponding to the environment;

determining, using probabilistic gating on a first data segment of the first sensor data and a portion of a first data segment of the second sensor data that correspond to the environment at a first time period, information indicative of parameters of one or more objects in the environment, wherein the probabilistic gating associates measurements within a given data segment of sensor data to respective objects in the environment;

modifying, using probabilistic dating on a second data segment of the first sensor data and a subsequent portion of the first data segment of the second sensor data that correspond to the environment at a second time period, the information indicative of the parameters of the one or more objects; and

providing the modified information indicative of the parameters of the one or more objects to one or more systems of the vehicle.

17. The non-transitory computer readable medium of claim 16 , further comprising:

based on the modified information indicative of the parameters of the one or more objects, providing one or more conceptual gates that substantially outline respective objects of the one or more objects in the environment to the one or more systems of the vehicle.

18. The non-transitory computer readable medium of claim 16 , further comprising:

receiving, from a third sensor at a third rate, third sensor data corresponding to the environment; and

wherein determining information indicative of parameters of one or more objects in the environment further comprises using at least a portion of a first data segment of the third sensor data.

Assignments (6)
CORRECTIVE ASSIGNMENT TO CORRECT THE THE REMOVAL OF THE INCORRECTLY RECORDED APPLICATION NUMBERS 14/149802 AND 15/419313 PREVIOUSLY RECORDED AT REEL: 44144 FRAME: 1. ASSIGNOR(S) HEREBY CONFIRMS THE CHANGE OF NAME. Recorded Mar 4, 2024
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 068092/0502 →
SUBMISSION TO CORRECT AN ERROR MADE IN A PREVIOUSLY RECORDED DOCUMENT THAT ERRONEOUSLY AFFECTS THE IDENTIFIED APPLICATIONS Recorded Nov 11, 2019
From: WAYMO LLC
To: WAYMO LLC
Reel/Frame 050978/0359 →
CHANGE OF NAME Recorded Oct 6, 2017
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 044144/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 24, 2017
From: GOOGLE INC.
To: WAYMO HOLDING INC.
Reel/Frame 042084/0741 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 24, 2017
From: WAYMO HOLDING INC.
To: WAYMO LLC
Reel/Frame 042085/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 6, 2015
From: ZHU, JIAJUN; AGARWAL, PRATIK
To: GOOGLE INC.
Reel/Frame 035983/0465 →