IP Library Granted Patent US 10,650,256
Granted Patent B2
US 10,650,256 · App. 15/490,682 · Granted May 12, 2020

Automatically perceiving travel signals

Inventors: Baoxing Qin (Singapore, SG); Aravindkumar Vijayalingam (Singapore, SG)
Assignee: nuTonomy Inc.
G06K9/00825B60W30/00G06K9/4604G06K9/6202G08G1/09623G08G1/096716G08G1/096725G08G1/096783G08G1/096791
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,650,256
App. No.
15/490,682
Granted
May 12, 2020
Kind
B2
Abstract

Among other things, one or more travel signals are identified by analyzing one or more images and data from sensors, classifying candidate travel signals into zero, one or more true and relevant travel signals, and estimating a signal state of the classified travel signals.

Claims (55)

1. An apparatus comprising:

(a) an image processor configured to receive an image derived from signals of a sensor and to apply signal processing to the image to identify a representation of a travel signal in the image as a candidate travel signal,

(b) a classifier configured to receive information from the image processor that identifies the candidate travel signal and to determine whether the candidate travel signal is relevant to a driving decision to be made for a vehicle at a current moment, wherein determining that the candidate travel signal is relevant to the driving decision to be made for the vehicle at the current moment comprises using prior information associated with the candidate travel signal to generate a prior image of a true travel signal during a prior traversal of a route corresponding to the candidate travel signal, wherein generating the prior image of the true travel signal comprises:

accessing a road map representing a neighborhood of the vehicle,

accessing information about travel signals in the neighborhood, the travel signals including the true travel signal, and

transforming the road map into the prior image in a field of view of the sensor, including synthesizing the true travel signal in the prior image,

(c) an estimator configured to estimate a signal state of the candidate travel signal in accordance with a determination that the candidate travel signal is relevant to the driving decision to be made for the vehicle at the current moment, and

(d) an output module to generate an alert or control a maneuver of the vehicle or both based on the estimated signal state.

2. The apparatus of claim 1 , in which the classifier is configured to classify the candidate travel signal as a true travel signal or not a true travel signal.

3. The apparatus of claim 1 , in which the image processor is configured to analyze pixels of the image based on saturation or lightness or both.

4. The apparatus of claim 1 , in which the image processor is configured to determine edges based on pixels and to generate a shape based on the edges.

5. The apparatus of claim 4 , in which the image processor is configured to identify the candidate travel signal based on one or more of the following criteria: edges, shapes, convexity, sizes, and solidness.

6. The apparatus of claim 1 , in which the classifier determines that the candidate travel signal is relevant to the driving decision to be made for the vehicle at the current moment based on one or more of the following: a previously identified travel signal, travel signal shapes, travel signal colors, travel signal positions, travel signal configurations, road networks, a location of the vehicle, and a route of a vehicle.

7. The apparatus of claim 1 , in which the prior information comprises one or more of the following: shapes, sizes, colors, locations, positions, and configurations.

8. The apparatus of claim 1 , in which the prior information includes a classification of the candidate travel signal as the true travel signal, wherein the classification comprises computing a classification score.

9. The apparatus of claim 8 , in which computing the classification score comprises computing a weighted sum of differences between measured data associated with the candidate travel signal and the prior information associated with the candidate travel signal.

10. The apparatus of claim 8 , in which computing the classification score is based on an algorithmic analysis on measured data associated with the candidate travel signal and the prior information.

11. The apparatus of claim 10 , in which the algorithmic analysis comprises (1) creating correspondences between the candidate travel signal and known true travel signals; (2) computing a likelihood score associated with the correspondences; and (3) iterating (1) and (2) using a different set of correspondences until an optimal likelihood score associated with an optimal set of correspondences is identified.

12. The apparatus of claim 1 , in which the estimator estimates the signal state by using state transition information.

13. The apparatus of claim 12 , in which the state transition information comprises colors, shapes, flashing patterns, or combinations of them.

14. The apparatus of claim 1 , in which the estimator estimates the signal state based on consistency of two or more travel signals.

15. The apparatus of claim 1 , in which the estimator estimates the signal state based on a position of a travel signal within a travel signal configuration.

16. The apparatus of claim 1 , in which the estimator estimates the signal state by temporal filtering based on a previously estimated signal state.

17. The apparatus of claim 1 , in which the output module generates a visual alert or an audio alert or both.

18. A vehicle comprising:

(a) an image processor configured to receive an image derived from signals of a sensor and to apply signal processing to the image to identify a representation of a travel signal in the image as a candidate travel signal,

(b) a classifier configured to receive information from the image processor that identifies the candidate travel signal and to determine whether the candidate travel signal is relevant to a driving decision to be made for the vehicle at a current moment, wherein determining that the candidate travel signal is relevant to the driving decision to be made for the vehicle at the current moment comprises using prior information associated with the candidate travel signal to generate a prior image of a true travel signal during a prior traversal of a route corresponding to the candidate travel signal, wherein generating the prior image of the true travel signal comprises:

accessing a road map representing a neighborhood of the vehicle,

accessing information about travel signals in the neighborhood, the travel signals including the true travel signal, and

transforming the road map into the prior image in a field of view of the sensor, including synthesizing the true travel signal in the prior image,

(c) an estimator configured to estimate a signal state of the candidate travel signal in accordance with a determination that the candidate travel signal is relevant to the driving decision to be made for the vehicle at the current moment, and

(d) an output module to generate an alert or control a maneuver of the vehicle or both based on the estimated signal state.

19. An apparatus comprising:

(a) an image processor configured to receive an image derived from signals of a sensor and to apply signal processing to the image to identify a representation of a travel signal in the image as a candidate travel signal,

(b) a classifier configured to receive information from the image processor that identifies the candidate travel signal and to determine whether the candidate travel signal is relevant to a driving decision to be made for a vehicle at a current moment, wherein determining that the candidate travel signal is relevant to the driving decision to be made for the vehicle at the current moment comprises using prior information associated with the candidate travel signal to generate a prior image of a true travel signal during a prior traversal of a route corresponding to the candidate travel signal, wherein the prior image comprises a transformation between a field of view of the sensor associated with the vehicle and a bird's eye view corresponding to the vehicle,

(c) an estimator configured to estimate a signal state of the candidate travel signal in accordance with a determination that the candidate travel signal is relevant to the driving decision to be made for the vehicle at the current moment, and

(d) an output module to generate an alert or control a maneuver of the vehicle or both based on the estimated signal state.

20. The apparatus of claim 19 , in which the classifier is configured to classify the candidate travel signal as a true travel signal or not a true travel signal.

21. The apparatus of claim 19 , in which the image processor is configured to analyze pixels of the image based on saturation or lightness or both.

22. The apparatus of claim 19 , in which the image processor is configured to determine edges based on pixels and to generate a shape based on the edges.

23. The apparatus of claim 22 , in which the image processor is configured to identify the candidate travel signal based on one or more of the following criteria: edges, shapes, convexity, sizes, and solidness.

24. The apparatus of claim 19 , in which the classifier determines that the candidate travel signal is relevant to the driving decision to be made for the vehicle at the current moment based on one or more of the following: a previously identified travel signal, travel signal shapes, travel signal colors, travel signal positions, travel signal configurations, road networks, a location of the vehicle, and a route of a vehicle.

25. The apparatus of claim 19 , in which the prior information comprises one or more of the following: shapes, sizes, colors, locations, positions, and configurations.

26. The apparatus of claim 19 , in which the prior information includes a classification of the candidate travel signal as a true travel signal, wherein the classification comprises computing a classification score.

27. The apparatus of claim 26 , in which computing the classification score comprises computing a weighted sum of differences between measured data associated with the candidate travel signal and the prior information associated with the candidate travel signal.

28. The apparatus of claim 26 , in which computing the classification score is based on an algorithmic analysis on measured data associated with the candidate travel signal and the prior information.

29. The apparatus of claim 28 , in which the algorithmic analysis comprises (1) creating correspondences between the candidate travel signal and known true travel signals; (2) computing a likelihood score associated with the correspondences; and (3) iterating (1) and (2) using a different set of correspondences until an optimal likelihood score associated with an optimal set of correspondences is identified.

30. The apparatus of claim 19 , in which the estimator estimates the signal state by using state transition information comprising colors, shapes, flashing patterns, or combinations of them.

31. The apparatus of claim 19 , in which the estimator estimates the signal state based on at least one of consistency of two or more travel signals, a position of a travel signal within a travel signal configuration, or temporal filtering based on a previously estimated signal state.

32. The apparatus of claim 19 , in which the output module generates a visual alert or an audio alert or both.

33. A vehicle comprising:

(a) an image processor configured to receive an image derived from signals of a sensor and to apply signal processing to the image to identify a representation of a travel signal in the image as a candidate travel signal,

(b) a classifier configured to receive information from the image processor that identifies the candidate travel signal and to determine whether the candidate travel signal is relevant to a driving decision to be made for a vehicle at a current moment, wherein determining that the candidate travel signal is relevant to the driving decision to be made for the vehicle at the current moment comprises using prior information associated with the candidate travel signal to generate a prior image of a true travel signal during a prior traversal of a route corresponding to the candidate travel signal, wherein the prior image comprises a transformation between a field of view of the sensor associated with the vehicle and a bird's eye view corresponding to the vehicle,

(c) an estimator configured to estimate a signal state of the candidate travel signal in accordance with a determination that the candidate travel signal is relevant to the driving decision to be made for the vehicle at the current moment, and

(d) an output module to generate an alert or control a maneuver of the vehicle or both based on the estimated signal state.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 2, 2020
From: MOTIONAL AD INC.
To: MOTIONAL AD LLC
Reel/Frame 053961/0619 →
CHANGE OF NAME Recorded Sep 25, 2020
From: NUTONOMY INC.
To: MOTIONAL AD INC.
Reel/Frame 053892/0220 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 19, 2017
From: QIN, BAOXING; VIJAYALINGAM, ARAVINDKUMAR
To: NUTONOMY INC.
Reel/Frame 042068/0080 →
Continuity (1)
Related Publication 20180300567A1 · Oct 18, 2018