IP Library Granted Patent US 12,018,958
Granted Patent B2
US 12,018,958 · App. 16/917,672 · Granted Jun 25, 2024

Systems and methods for inferring information about stationary elements based on semantic relationships

Inventors: Ritwik Subir Das (Mountain View, CA); Kris Richard Efland (Lafayette, CA); Nadha Nafeeza Gafoor (Sunnyvale, CA); Nastaran Ghadar (Pleasanton, CA); Meng Tao (Redwood City, CA)
Assignee: Lyft, Inc.
G01C21/3602G01C21/3492G01C21/3833G01C21/3896G06F18/214G06V20/582G06V20/584G06V20/588
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,018,958
App. No.
16/917,672
Granted
Jun 25, 2024
Kind
B2
Abstract

Examples disclosed herein may involve a computing system configured to (i) identify a stationary element in a real-world environment for which to infer information, (ii) detect a semantic relationship between the stationary element and one or more other stationary elements in the real-world environment, (iii) based on the detected semantic relationship, infer information about the stationary element, (iv) include the inferred information about the stationary element within a set of information that describes the stationary element.

Claims (62)

1. A computer-implemented method comprising:

after one or more sensor-equipped vehicles have traversed a real-world environment and captured sensor data that is representative of the real-world environment, performing an analysis of the sensor data;

based on the analysis of the sensor data, detecting a stationary element in a real-world environment;

accessing map data for the real-world environment;

evaluating whether the detected stationary element is encoded within the accessed map data for the real-world environment and thereby determining that the detected stationary element is not encoded within the accessed map data for the real-world environment; and

in response to determining that the detected stationary element is not encoded within the accessed map data for the real-world environment:

deriving a first set of information about the detected stationary element based on an analysis of sensor data that is representative of the detected stationary element;

detecting a semantic relationship between the detected stationary element and one or more other stationary elements in the real-world environment based on an analysis of one or both of (i) the sensor data captured by one or more sensor-equipped vehicles while operating in the real-world environment or (ii) the accessed map data for the real-world environment;

based on the detected semantic relationship, inferring a second set of information about the detected stationary element; and

combining the first and second sets of information about the detected stationary element into a combined set of information that describes the detected stationary element.

2. The computer-implemented method of claim 1 , wherein the detected stationary element comprises a traffic light and the one or more other stationary elements comprise one or both of (i) a traffic sign or (ii) a traffic lane.

3. The computer-implemented method of claim 2 , wherein the inferred information comprises one or more of (i) signal-face information for the traffic light, (ii) lane-control information for the traffic light, or (iii) traffic-rule information for the traffic light.

4. The computer-implemented method of claim 1 , wherein the detected stationary element comprises a traffic light, and wherein detecting the semantic relationship between the traffic light and the one or more other stationary elements comprises:

detecting a traffic sign of a given type that is located in a given direction relative to the traffic light and within a threshold distance from the traffic light.

5. The computer-implemented method of claim 1 , wherein the detected stationary element comprises a traffic lane, and wherein detecting the semantic relationship between the traffic lane and the one or more other stationary elements comprises:

detecting a traffic sign of a given type that is located adjacent to the traffic lane.

6. The computer-implemented method of claim 1 , wherein inferring the second set of information about the detected stationary element comprises:

deriving information about the one or more other stationary elements in the real-world environment; and

using the derived information about the one or more other stationary elements as a basis for inferring the information about the detected stationary element.

7. The computer-implemented method of claim 6 , wherein the one or more other stationary elements in the real-world environment comprise a traffic sign, and wherein the derived information about the traffic sign comprises one or more of (i) a classification of the traffic sign or (ii) an indication of text that is displayed on the traffic sign.

8. The computer-implemented method of claim 1 , further comprising:

encoding the combined set of information that describes the detected stationary element into a map for the real-world environment.

9. The computer-implemented method of claim 1 , further comprising:

using the combined set of information that describes the detected stationary element as training data for a machine-learning model that is configured to predict information about corresponding stationary elements of a same type as the detected stationary element.

10. The computer-implemented method of claim 9 , wherein the information about the corresponding stationary elements comprises information indicating that a semantic relationship exists between the corresponding stationary elements and one or more other stationary elements.

11. The computer-implemented method of claim 1 , further comprising:

updating a map for the real-world environment to include the combined set of information that describes the detected stationary element; and

based on the combined set of information that describes the detected stationary element in the updated map, generating a route for a vehicle to follow in the real-world environment.

12. A non-transitory computer-readable medium comprising program instructions stored thereon that are executable to cause a computing system to:

after one or more sensor-equipped vehicles have traversed a real-world environment and captured sensor data that is representative of the real-world environment, perform an analysis of the sensor data;

based on the analysis of the sensor data, detect a stationary element in a real-world environment;

access map data for the real-world environment;

evaluate whether the detected stationary element is encoded within the accessed map data for the real-world environment and thereby determine that the detected stationary element is not encoded within the accessed map data for the real-world environment; and

in response to determining that the detected stationary element is not encoded within the accessed map data for the real-world environment:

derive a first set of information about the detected stationary element based on an analysis of sensor data that is representative of the detected stationary element;

detect a semantic relationship between the detected stationary element and one or more other stationary elements in the real-world environment based on an analysis of one or both of (i) the sensor data captured by one or more sensor-equipped vehicles while operating in the real-world environment or (ii) the accessed map data for the real-world environment;

based on the detected semantic relationship, infer a second set of information about the detected stationary element; and

combine the first and second sets of information about the detected stationary element into a combined set of information that describes the detected stationary element.

13. The computer-readable medium of claim 12 , wherein the detected stationary element comprises a traffic light and the one or more other stationary elements comprise one or both of (i) a traffic sign or (ii) a traffic lane.

14. The computer-readable medium of claim 13 , wherein the inferred information comprises one or more of (i) signal-face information for the traffic light, (ii) lane-control information for the traffic light, or (iii) traffic-rule information for the traffic light.

15. The computer-readable medium of claim 12 , wherein the detected stationary element comprises a traffic light, and wherein detecting the semantic relationship between the traffic light and the one or more other stationary elements comprises:

detecting a traffic sign of a given type that is located in a given direction relative to the traffic light and within a threshold distance from the traffic light.

16. The computer-readable medium of claim 12 , wherein the detected stationary element comprises a traffic lane, and wherein detecting the semantic relationship between the traffic lane and the one or more other stationary elements comprises:

detecting a traffic sign of a given type that is located adjacent to the traffic lane.

17. The computer-readable medium of claim 12 , wherein the computer-readable medium further comprises program instructions stored thereon that are executable to cause the computing system to:

encode the combined set of information that describes the detected stationary element into a map for the real-world environment.

18. The computer-readable medium of claim 12 , wherein the computer-readable medium further comprises program instructions stored thereon that are executable to cause the computing system to:

update a map for the real-world environment to include the combined set of information that describes the detected stationary element; and

based on the combined set of information that describes the detected stationary element in the updated map, generate a route for a vehicle to follow in the real-world environment.

19. A computing system comprising:

at least one processor;

a non-transitory computer-readable medium; and

program instructions stored on the non-transitory computer-readable medium that are executable by the at least one processor such that the computing system is capable of:

after one or more sensor-equipped vehicles have traversed a real-world environment and captured sensor data that is representative of the real-world environment, performing an analysis of the sensor data;

based on the analysis of the sensor data, detecting a stationary element in a real-world environment;

accessing map data for the real-world environment;

evaluating whether the detected stationary element is encoded within the accessed map data for the real-world environment and thereby determining that the detected stationary element is not encoded within the accessed map data for the real-world environment; and

in response to determining that the detected stationary element is not encoded within the accessed map data for the real-world environment:

deriving a first set of information about the detected stationary element based on an analysis of sensor data that is representative of the detected stationary element;

detecting a semantic relationship between the detected stationary element and one or more other stationary elements in the real-world environment based on an analysis of one or both of (i) the sensor data captured by one or more sensor-equipped vehicles while operating in the real-world environment or (ii) the accessed map data for the real-world environment;

based on the detected semantic relationship, inferring a second set of information about the detected stationary element; and

combining the first and second sets of information about the detected stationary element into a combined set of information that describes the detected stationary element.

Assignments (2)
SECURITY INTEREST Recorded Nov 3, 2022
From: LYFT, INC.
To: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 061880/0237 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 20, 2020
From: DAS, RITWIK SUBIR; EFLAND, KRIS RICHARD; GAFOOR, NADHA NAFEEZA; GHADAR, NASTARAN; TAO, MENG
To: LYFT, INC.
Reel/Frame 053548/0344 →