IP Library Granted Patent US 11,010,615
Granted Patent B2
US 11,010,615 · App. 15/812,636 · Granted May 18, 2021

Rendering a situational-awareness view in an autonomous-vehicle environment

Inventors: Robert Earl Rasmusson, Jr. (San Francisco, CA); Taggart Matthiesen (Kentfield, CA); Craig Dehner (San Francisco, CA); Linda Dong (San Francisco, CA); Frank Taehyun Yoo (San Carlos, CA); Karina van Schaardenburg (San Francisco, CA); John Tighe (San Francisco, CA); Matt Vitelli (San Francisco, CA); Jisi Guo (San Francisco, CA); Eli Guerron (San Francisco, CA)
Assignee: Lyft, Inc.
G06K9/00791B62D15/0285G01C21/365G01C21/3638G05D1/0044G05D1/0088G05D1/0212G05D1/0246G05D1/0274G06K9/6267G06T7/20B60W2554/00G05D2201/0212G05D2201/0213G06T2207/30252G06T2207/30256G06T2207/30261
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,010,615
App. No.
15/812,636
Granted
May 18, 2021
Kind
B2
Abstract

In one embodiment, a method includes receiving autonomous-vehicle sensor data from a sensor array of an autonomous vehicle. The autonomous-vehicle sensor data indicates an object in an external environment of the autonomous vehicle. The method further includes determining a confidence score for a classification of the object. The method further includes determining an object graphic with a level of detail to visually represent the object. The level of detail of the object graphic is based on the confidence score. The method further includes providing for display a visual representation the object graphic.

Claims (63)

1. A method comprising, by one or more computing devices:

receiving first sensor data from a sensor array of a vehicle while traveling on a road, wherein the first sensor data is associated with a distance between the vehicle and an object along the road in an environment external of the vehicle;

determining a confidence score for a classification of the object based on the first sensor data associated with the distance between the object and the vehicle;

determining a level of visual detail to visually represent the object as a first object graphic based on the determined confidence score for the classification of the object;

providing for display a visual representation of the first object graphic rendered at the level of visual detail and a road graphic associated with the road;

subsequent to providing for display the visual representation of the first object graphic and the road graphic, adjusting the confidence score for the classification of the object based on second sensor data, received from the sensor array of the vehicle, that is associated with a second distance between the object and the vehicle;

adjusting the level of visual detail to visually represent the object as a second object graphic based on the adjusted confidence score for the classification of the object; and

providing for display a visual representation of the second object graphic rendered at the adjusted level of visual detail and the road graphic associated with the road.

2. The method of claim 1 , wherein determining the confidence score for the classification of the object comprises:

comparing the received first sensor data with a pre-determined pattern that corresponds to the classification; and

determining a difference between the pre-determined pattern and the received first sensor data.

3. The method of claim 1 , wherein the first object graphic and the second object graphic share one or more features representative of the object.

4. The method of claim 3 , wherein

wherein the second object graphic has more features representative of the object than the first object graphic.

5. The method of claim 1 , wherein the first sensor data or the second sensor data comprises a list of coordinate points corresponding to one or more objects within a threshold distance of the vehicle.

6. The method of claim 1 , further comprising:

determining, based on the determined confidence score and the adjusted confidence score for the classification of the object, a first opacity for the first object graphic and a second opacity for the second object graphic, respectively.

7. The method of claim 1 , wherein the first sensor data or the second sensor data comprises:

a labeled point set corresponding to each object represented by an object graphic; and

a confidence score for the labeled point set.

8. The method of claim 1 , wherein the adjusted confidence score is greater than the determined confidence score.

9. The method of claim 1 , wherein the first sensor data or the second sensor data further indicates a plurality of additional objects in the environment external of the vehicle and is associated with a distance between the vehicle and each additional object of the plurality of additional objects, wherein the method further comprises:

determining, for each additional object of the plurality of additional objects, a level of visual detail to visually represent the additional object based on a determined distance between the object and the vehicle; and

selecting an object graphic for each additional object of the plurality of additional objects, wherein a first additional object closer to the vehicle is represented by an object graphic that has a higher level of visual detail than an object graphic representing a second additional object farther from the vehicle.

10. The method of claim 1 , further comprising:

based on the second sensor data, determining that the object is moving; and

providing instructions to render the visual representation of the second object graphic with a graphical indication that the object is moving.

11. The method of claim 1 , wherein at least some of the first sensor data or the second sensor data is generated by one or more sensors of the vehicle.

12. The method of claim 1 , wherein the method further comprises:

determining, based on the first sensor data or the second sensor data, an overlay graphic that provides information associated with one or more of a state of the vehicle, a current ride of the vehicle, a current passenger of the vehicle, or a current route of the vehicle,

wherein the visual representation provided for display further comprises the overlay graphic.

13. A system comprising: one or more processors and one or more computer-readable non-transitory storage media coupled to one or more of the processors, the one or more computer-readable non-transitory storage media comprising instructions operable when executed by the one or more processors to cause the system to perform operations comprising:

receiving first sensor data from a sensor array of a vehicle while traveling on a road, wherein the first sensor data is associated with a distance between the vehicle and an object along the road and in an environment external of the vehicle;

determining a confidence score for a classification of the object based on the first sensor data associated with the distance between the object and the vehicle;

determining a level of visual detail to visually represent the object as a first object graphic based on the determined confidence score for the classification of the object;

providing for display a visual representation of the first object graphic rendered at the level of visual detail and a road graphic associated with the road;

subsequent to providing for display the visual representation of the first object graphic and the road graphic, adjusting the confidence score for the classification of the object based on second sensor data, received from the sensor array of the vehicle, that is associated with a second distance between the object and the vehicle;

adjusting the level of visual detail to visually represent the object as a second object graphic based on the adjusted confidence score for the classification of the object; and

providing for display a visual representation of the second object graphic rendered at the adjusted level of visual detail and the road graphic associated with the road.

14. The system of claim 13 , wherein determining the confidence score for the classification of the object comprises:

comparing the received first sensor data with a pre-determined pattern that corresponds to the classification; and

determining a difference between the pre-determined pattern and the received first sensor data.

15. One or more computer-readable non-transitory storage media including instructions that, when executed by one or more processors, are configured to cause the one or more processors to perform operations comprising:

receiving first sensor data from a sensor array of a vehicle while traveling on a road, wherein the first sensor data is associated with a distance between the vehicle and an object along the road in an environment external of the vehicle;

determining a confidence score for a classification of the object based on the first sensor data associated with the distance between the object and the vehicle;

determining a level of visual detail to visually represent the object as a first object graphic based on the determined confidence score for the classification of the object;

providing for display a visual representation of the first object graphic rendered at the level of visual detail and a road graphic associated with the road;

subsequent to providing for display the visual representation of the first object graphic and the road graphic, adjusting the confidence score for the classification of the object based on second sensor data, received from the sensor array of the vehicle, that is associated with a second distance between the object and the vehicle;

adjusting the level of visual detail to visually represent the object as a second object graphic based on the adjusted confidence score for the classification of the object; and

providing for display a visual representation of the second object graphic rendered at the adjusted level of visual detail and the road graphic associated with the road.

16. The computer-readable non-transitory storage media of claim 15 , wherein determining the confidence score for the classification of the object comprises:

comparing the received first sensor data with a pre-determined pattern that corresponds to the classification; and

determining a difference between the pre-determined pattern and the received first sensor data.

17. The method of claim 1 , wherein the level of visual detail to visually represent the object as a first object graphic is associated with a plurality of levels of visual detail, each level of visual detail being associated with a threshold score; and

wherein determining the level of visual detail to visually represent the object as a first object graphic based on the determined confidence score for the classification of the object comprises:

determining that the confidence score for the classification satisfies a particular threshold score; and

identifying the level of visual detail associated with the particular threshold score.

18. The method of claim 1 , wherein adjusting the level of visual detail to visually represent the object as a second object graphic based on the adjusted confidence score for the classification of the object comprises:

increasing the level of visual detail to visually represent the object based on determining that the adjusted confidence score for the classification of the object is greater than the determined confidence score for the classification of the object.

19. The method of claim 1 , wherein the first object graphic and the second object graphic are computer-generated graphical representations of the object.

20. The method of claim 1 , further comprising:

determining, based on the first sensor data, movement of the object relative to the vehicle,

wherein determining the level of visual detail to visually represent the object as the first object graphic is further based on the determined movement of the object relative to the vehicle while the vehicle is traveling on the road.

Assignments (2)
SECURITY INTEREST Recorded Nov 3, 2022
From: LYFT, INC.
To: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 061880/0237 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 25, 2018
From: RASMUSSON, ROBERT EARL; MATTHIESEN, TAGGART; DEHNER, CRAIG; DONG, LINDA; YOO, FRANK TAEHYUN; VAN SCHAARDENBURG, KARINA; TIGHE, JOHN; VITELLI, MATT; GUO, JISI; GUERRON, ELI
To: LYFT, INC.
Reel/Frame 046196/0331 →
Continuity (2)
Provisional Application 62422025 · Nov 14, 2016
Related Publication 20180137373A1 · May 17, 2018
Cited By (5)
US 12,205,030 US 12,235,120 US 12,235,121 US 12,264,933 US 12,487,354