IP Library Granted Patent US 10,890,915
Granted Patent B2
US 10,890,915 · App. 16/463,236 · Granted Jan 12, 2021

Solution path overlay interfaces for autonomous vehicles

Inventors: Liam Pedersen (San Francisco, CA); Maarten Sierhuis (San Francisco, CA); Hans Utz (Campbell, CA); Mauro Della Penna (San Francisco, CA); Terrence Fong (Moffett Field, CA); Mark Allan (Campbell, CA); Maria Bualat (San Jose, CA); Eric Schafer (Kentfield, CA)
Assignees: Nissan North America, Inc.; United States of America as Represented by the Administrator of NASA
G05D1/0214G01C21/3676G05D1/0044G05D2201/0213
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,890,915
App. No.
16/463,236
Granted
Jan 12, 2021
Kind
B2
Abstract

Methods and systems for generating a solution path overlay interface to transmit a solution path are described. The disclosed technology includes receiving vehicle data and external data from a vehicle. The vehicle data includes a vehicle location and a vehicle destination, and the external data includes a location and a movement path for each of a plurality of external objects. A solution path is determined between the vehicle location and the vehicle destination, wherein the solution path does not intersect with the plurality of external objects. A solution path overlay interface is generated that includes the vehicle traveling the solution path and at least some of the plurality of external objects. The solution path overlay interface is outputted for display that is configured to receive a command from an operator which results in an updated solution path that is transmitted to the vehicle for execution.

Claims (65)

1. A method, comprising:

receiving, by a solution path overlay system, vehicle data and external data from a vehicle control system of a vehicle, wherein the vehicle data is associated with the vehicle and includes a vehicle location and a vehicle destination, and the external data includes a location and a movement path for each of a plurality of external objects;

determining, by the solution path overlay system, a solution path between the vehicle location and the vehicle destination using the vehicle data and the external data;

generating, by the solution path overlay system, a solution path overlay interface that includes an indicator of the solution path, an indicator of a solution path overlay that corresponds to the solution path, an indicator of a roadway, a representation of the vehicle traversing along the solution path via the roadway, and an indicator for each of at least some of the plurality of external objects, the solution path overlay representing outer edges of the solution path based on a width of the vehicle;

outputting, by the solution path overlay system, the solution path overlay interface for display to an operator of the solution path overlay interface;

in response to detecting a change associated with the movement path of any of the plurality of external objects, receiving, by the solution path overlay system, a command from the operator of the solution path overlay interface, the command including a drawing gestured directly onto the solution path overlay interface and an indication of one or more stop points;

updating, by the solution path overlay system, the solution path overlay interface based on the received command to provide an updated solution path and an updated solution path overlay that corresponds to the updated solution path; and

transmitting, by the solution path overlay system, the updated solution path and the updated solution path overlay to the vehicle for execution to navigate to the vehicle destination.

2. The method of claim 1 , wherein any of the vehicle data and the external data includes sensor data from any of an optical sensor, a light detection and ranging sensor (LIDAR), a motion sensor, and an audio sensor.

3. The method of claim 2 , further comprising:

determining, by the solution path overlay system, a surface state within a predetermined distance of the vehicle based on the sensor data; and

generating, by the solution path overlay system, a surface image including a surface state indication based on the surface state, wherein the solution path overlay interface includes the surface state indication.

4. The method of claim 2 , further comprising:

determining, by the solution path overlay system, traffic conditions using the sensor data and movement paths of the plurality of external objects within a predetermined area, wherein a vehicle route of the vehicle is determined using the traffic conditions, wherein the vehicle route provides navigation data for the vehicle between the vehicle location and the vehicle destination.

5. The method of claim 4 , further comprising:

determining, by a solution path overlay system, a direction of regulated traffic flow based on any of traffic flow data and traffic flow indicators within the predetermined area, wherein the traffic flow data includes authorized traffic flow patterns for the plurality of external objects, wherein the solution path overlay interface includes an indication of the direction of regulated traffic flow.

6. The method of claim 1 , wherein the vehicle location is based on map data that includes a geographical area of the vehicle and of the plurality of external objects, further comprising:

displaying the updated solution path to an operator of the vehicle.

7. The method of claim 1 , further comprising:

determining, by the solution path overlay system, identification data of the plurality of external objects based on any of the vehicle data and the external data;

determining, by the solution path overlay system, mobility states of the plurality of external objects, wherein the mobility states include moveable states and immoveable states; and

updating, by the solution path overlay system and based on the mobility states, the solution path overlay interface to indicate the moveable states and the immoveable states of the plurality of external objects.

8. The method of claim 7 , wherein the identification data comprises any of pedestrians, vehicles, traffic light signals, traffic signage, pavement markings, buildings, primary roadways, and secondary roadways.

9. The method of claim 7 , further comprising:

determining, by the solution path overlay system for each portion of the solution path, a buffer distance between the vehicle and each of the plurality of external objects, wherein the determining of the solution path further comprises dynamically maintaining the buffer distance between the vehicle and each of the plurality of external objects.

10. A solution path overlay system comprising:

a memory; and

a processor configured to execute instructions stored in the memory to:

receive vehicle data and external data from a vehicle control system of a vehicle, wherein the vehicle data is associated with the vehicle and includes a vehicle location and a vehicle destination, and the external data includes a location and a movement path for each of a plurality of external objects;

determine a solution path between the vehicle location and the vehicle destination using the vehicle data and the external data;

generate a solution path overlay interface that includes an indicator of the solution path, an indicator of a solution path overlay that corresponds to the solution path, an indicator of a roadway, a representation of the vehicle traversing along the solution path via the roadway, and an indicator for each of at least some of the plurality of external objects, the solution path overlay representing outer edges of the solution path based on a width of the vehicle;

output the solution path overlay interface for display to an operator of the solution path overlay interface;

in response to detecting a change associated with the movement path of any of the plurality of external objects, receive a command from the operator of the solution path overlay interface, the command including a drawing gestured directly onto the solution path overlay interface and an indication of one or more stop points;

update the solution path overlay interface based on the received command to provide an updated solution path and an updated solution path overlay that corresponds to the updated solution path; and

transmit the updated solution path and the updated solution path overlay to the vehicle for execution to navigate to the vehicle destination.

11. The solution path overlay system of claim 10 , wherein any of the vehicle data and the external data includes sensor data from any of an optical sensor, a light detection and ranging sensor (LIDAR), a motion sensor, and an audio sensor.

12. The solution path overlay system of claim 11 , wherein the processor is further configured to execute instructions stored in the memory to:

determine a surface state within a predetermined distance of the vehicle based on the sensor data; and

generate a surface image including a surface state indication based on the surface state, wherein the solution path overlay interface includes the surface state indication.

13. The solution path overlay system of claim 11 , wherein the processor is further configured to execute instructions stored in the memory to:

determine traffic conditions using the sensor data and movement paths of the plurality of external objects within a predetermined area, wherein a vehicle route is determined using the traffic conditions, wherein the vehicle route provides navigation data for the vehicle between the vehicle location and the vehicle destination.

14. The solution path overlay system of claim 11 , wherein the vehicle location is based on map data that includes a geographical area of the vehicle and of the plurality of external objects, the processor further configured to execute instructions stored in the memory to:

display the updated solution path to an operator of the vehicle.

15. The solution path overlay system of claim 11 , wherein the processor is further configured to execute instructions stored in the memory to:

determine identification data of the plurality of external objects based on any of the vehicle data and the external data;

determine mobility states of the plurality of external objects, wherein the mobility states include moveable states and immoveable states; and

update, based on the mobility states, the solution path overlay interface to indicate the moveable states and the immoveable states of the plurality of external objects.

16. A non-transitory computer-readable storage medium including program instructions executable by one or more processors of a solution path overlay system that, when executed, cause the one or more processors to perform operations, the operations comprising:

receiving vehicle data and external data from a vehicle control system of a vehicle, wherein the vehicle data is associated with the vehicle and includes a vehicle location and a vehicle destination, and the external data includes a location and a movement path for each of a plurality of external objects;

determining a solution path between the vehicle location and the vehicle destination using the vehicle data and the external data;

generating a solution path overlay interface that includes an indicator of the solution path, an indicator of a solution path overlay that corresponds to the solution path, an indicator of a roadway, a representation of the vehicle traversing along the solution path via the roadway, and an indicator for each of at least some of the plurality of external objects, the solution path overlay representing outer edges of the solution path based on a width of the vehicle;

outputting, by the solution path overlay system, the solution path overlay interface for display to an operator of the solution path overlay interface;

in response to detecting a change associated with the movement path of any of the plurality of external objects, receiving, by the solution path overlay system, a command from the operator of the solution path overlay interface, the command including a drawing gestured directly onto the solution path overlay interface and an indication of one or more stop points;

updating, by the solution path overlay system, the solution path overlay interface based on the received command to provide an updated solution path and an updated solution path overlay that corresponds to the updated solution path; and

transmitting the updated solution path and the updated solution path overlay to the vehicle for execution to navigate to the vehicle destination.

17. The non-transitory computer-readable storage medium of claim 16 , wherein any of the vehicle data and the external data includes sensor data from any of an optical sensor, a light detection and ranging sensor (LIDAR), a motion sensor, and an audio sensor.

18. The non-transitory computer-readable storage medium of claim 17 , the operations further comprising:

determining a surface state within a predetermined distance of the vehicle based on the sensor data; and

generating a surface image including a surface state indication based on the surface state, wherein the solution path overlay interface includes the surface state indication.

19. The non-transitory computer-readable storage medium of claim 17 , the operations further comprising:

determining traffic conditions using the sensor data and movement paths of the plurality of external objects within a predetermined area, wherein a vehicle route is determined using the traffic conditions, wherein the vehicle route provides navigation data for the vehicle between the vehicle location and the vehicle destination.

20. The non-transitory computer-readable storage medium of claim 17 , the operations further comprising:

determining identification data of the plurality of external objects based on any of the vehicle data and the external data;

determining mobility states of the plurality of external objects, wherein the mobility states include moveable states and immoveable states; and

updating, based on the mobility states, the solution path overlay interface to indicate the moveable states and the immoveable states of the plurality of external objects.

Assignments (5)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 26, 2021
From: NISSAN NORTH AMERICA, INC.
To: NISSAN MOTOR CO., LTD.
Reel/Frame 056038/0293 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 22, 2019
From: BUALAT, MARIA
To: UNITED STATES OF AMERICA AS REPRESENTED BY THE ADMINISTRATOR OF NASA
Reel/Frame 049258/0114 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 22, 2019
From: PEDERSEN, LIAM; SIERHUIS, MAARTEN; UTZ, HANS
To: NISSAN NORTH AMERICA, INC.
Reel/Frame 049258/0177 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 22, 2019
From: ALLAN, MARK; SCHAFER, ERIC
To: UNITED STATES OF AMERICA AS REPRESENTED BY THE ADMINISTRATOR OF NASA
Reel/Frame 049258/0219 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 22, 2019
From: FONG, TERRENCE
To: UNITED STATES OF AMERICA AS REPRESENTED BY THE ADMINISTRATOR OF NASA
Reel/Frame 049258/0258 →
Continuity (2)
Provisional Application 62430864 · Dec 6, 2016
Related Publication 20190339701A1 · Nov 7, 2019
Cited By (1)
US 12,715,356