IP Library Granted Patent US 10,643,084
Granted Patent B2
US 10,643,084 · App. 15/490,616 · Granted May 5, 2020

Automatically perceiving travel signals

Inventors: Baoxing Qin (Singapore, SG); Aravindkumar Vijayalingam (Singapore, SG)
Assignee: nuTonomy Inc.
G06K9/00818B60W30/00G06K9/4604G06K9/4652G06K9/481G06K9/6215G06K9/6278G08G1/09623G08G1/09626
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,643,084
App. No.
15/490,616
Granted
May 5, 2020
Kind
B2
Abstract

Among other things, one or more travel signals are identified by analyzing one or more images and data from sensors, classifying candidate travel signals into zero, one or more true and relevant travel signals, and estimating a signal state of the classified travel signals.

Claims (41)

1. A method comprising:

(a) receiving an image of a field of view of a sensor associated with a vehicle,

(b) identifying a candidate travel signal in the image,

(c) determining whether the candidate travel signal is relevant to a driving decision to be made for the vehicle at a current moment, wherein determining that the candidate travel signal is relevant to the driving decision to be made for the vehicle at the current moment comprises using prior information associated with the candidate travel signal to generate a prior image of a true travel signal during a prior traversal of a route corresponding to the candidate travel signal, wherein generating the prior image of the true travel signal comprises:

accessing a road map representing a neighborhood of the vehicle,

accessing information about travel signals in the neighborhood, the travel signals including the true travel signal, and

transforming the road map into the prior image in a field of view of the sensor, including synthesizing the true travel signal in the prior image, and

(d) in accordance with a determination that the candidate travel signal is relevant to the driving decision to be made for the vehicle at the current moment, alerting a driver of the vehicle of a signal state of the candidate travel signal.

2. The method of claim 1 , in which the identifying the candidate travel signal comprises analyzing pixels of the image based on saturation or lightness or both.

3. The method of claim 1 , in which the identifying the candidate travel signal comprises determining edges based on pixels and generating a shape based on the edges.

4. The method of claim 3 , in which the identifying the candidate travel signal is based on one or more of the following criteria: edges, shapes, convexity, sizes, and solidness.

5. The method of claim 1 , in which determining that the candidate travel signal is relevant to a driving decision to be made for the vehicle at a current moment is based on one or more of the following: a previously identified travel signal, travel signal shapes, travel signal colors, travel signal positions, travel signal configurations, road networks, a location of the vehicle, and a route of the vehicle.

6. The method of claim 1 , in which the prior information comprises one or more of the following: shapes, sizes, colors, locations, positions, and configurations.

7. The method of claim 1 , in which the prior information includes a classification of the candidate travel signal as the true travel signal, wherein the classification comprises computing a classification score.

8. The method of claim 7 , in which the classification score comprises a weighted sum of differences between measured data associated with the candidate travel signal and prior information associated with the candidate travel signal.

9. The method of claim 7 , in which the classification score is computed based on an algorithmic analysis on measured data associated with the candidate travel signal and the prior information.

10. The method of claim 9 , in which the algorithmic analysis comprises (1) creating correspondences between the candidate travel signal and known true travel signals; (2) computing a likelihood score associated with the correspondences; and (3) iterating (1) and (2) using a different set of correspondences until an optimal likelihood score associated with an optimal set of correspondences is identified.

11. The method of claim 1 , in which the determining that the candidate travel signal is relevant to the driving decision to be made for the vehicle at the current moment comprises estimating the signal state using state transition information.

12. The method of claim 11 , in which the state transition information comprises colors, shapes, flashing patterns, or combinations of them.

13. The method of claim 1 , in which the determining that the candidate travel signal is relevant to the driving decision to be made for the vehicle at the current moment comprises estimating the signal state based on consistency of two or more travel signals.

14. The method of claim 1 , in which the determining that the candidate travel signal is relevant to the driving decision to be made for the vehicle at the current moment comprises estimating the signal state based on a position of a travel signal within a travel signal configuration.

15. The method of claim 1 , in which the determining that the candidate travel signal is relevant to the driving decision to be made for the vehicle at the current moment comprises estimating the signal state using temporal filtering based on a previously estimated signal state.

16. A method comprising:

(a) receiving an image of a field of view of a sensor associated with a vehicle,

(b) identifying a candidate travel signal in the image,

(c) determining whether the candidate travel signal is relevant to a driving decision to be made for the vehicle at a current moment, wherein determining that the candidate travel signal is relevant to the driving decision to be made for the vehicle at the current moment comprises using prior information associated with the candidate travel signal to generate a prior image of a true travel signal during a prior traversal of a route corresponding to the candidate travel signal, wherein the prior image comprises a transformation between a field of view of the sensor associated with the vehicle and a bird's eye view corresponding to the vehicle, and

(d) in accordance with a determination that the candidate travel signal is relevant to the driving decision to be made for the vehicle at the current moment, alerting a driver of the vehicle of a signal state of the candidate travel signal.

17. The method of claim 16 , in which the identifying the candidate travel signal comprises analyzing pixels of the image based on saturation or lightness or both.

18. The method of claim 16 , in which the identifying the candidate travel signal comprises determining edges based on pixels and generating a shape based on the edges.

19. The method of claim 18 , in which the identifying the candidate travel signal is based on one or more of the following criteria: edges, shapes, convexity, sizes, and solidness.

20. The method of claim 16 , in which determining that the candidate travel signal is relevant to a driving decision to be made for the vehicle at a current moment is based on one or more of the following: a previously identified travel signal, travel signal shapes, travel signal colors, travel signal positions, travel signal configurations, road networks, a location of the vehicle, and a route of the vehicle.

21. The method of claim 16 , in which the prior information comprises one or more of the following: shapes, sizes, colors, locations, positions, and configurations.

22. The method of claim 16 , in which the prior information includes a classification of the candidate travel signal as the true travel signal, wherein the classification comprises computing a classification score.

23. The method of claim 22 , in which the classification score comprises a weighted sum of differences between measured data associated with the candidate travel signal and prior information associated with the candidate travel signal.

24. The method of claim 22 , in which the classification score is computed based on an algorithmic analysis on measured data associated with the candidate travel signal and the prior information.

25. The method of claim 24 , in which the algorithmic analysis comprises (1) creating correspondences between the candidate travel signal and known true travel signals; (2) computing a likelihood score associated with the correspondences; and (3) iterating (1) and (2) using a different set of correspondences until an optimal likelihood score associated with an optimal set of correspondences is identified.

26. The method of claim 16 , in which the determining that the candidate travel signal is relevant to the driving decision to be made for the vehicle at the current moment comprises estimating the signal state using state transition information.

27. The method of claim 26 , in which the state transition information comprises colors, shapes, flashing patterns, or combinations of them.

28. The method of claim 16 , in which the determining that the candidate travel signal is relevant to the driving decision to be made for the vehicle at the current moment comprises estimating the signal state based on consistency of two or more travel signals.

29. The method of claim 16 , in which the determining that the candidate travel signal is relevant to the driving decision to be made for the vehicle at the current moment comprises estimating the signal state based on a position of a travel signal within a travel signal configuration.

30. The method of claim 16 , in which the determining that the candidate travel signal is relevant to the driving decision to be made for the vehicle at the current moment comprises estimating the signal state using temporal filtering based on a previously estimated signal state.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 2, 2020
From: MOTIONAL AD INC.
To: MOTIONAL AD LLC
Reel/Frame 053961/0619 →
CHANGE OF NAME Recorded Sep 25, 2020
From: NUTONOMY INC.
To: MOTIONAL AD INC.
Reel/Frame 053892/0220 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 19, 2017
From: QIN, BAOXING; VIJAYALINGAM, ARAVINDKUMAR
To: NUTONOMY INC.
Reel/Frame 042068/0041 →
Continuity (1)
Related Publication 20180300565A1 · Oct 18, 2018