IP Library Granted Patent US 10,839,473
Granted Patent B2
US 10,839,473 · App. 16/349,486 · Granted Nov 17, 2020

Autonomous vehicle monitoring using generated interfaces

Inventors: Liam Pedersen (San Francisco, CA); Maarten Sierhuis (San Francisco, CA); Hans Utz (Campbell, CA); Mauro Della Penna (San Francisco, CA); Jeffrey Bradshaw (Nampa, ID); Matthew Johnson (Pensacola, FL); Michael Vignati (Pensacola, FL); Lawrence Bunch (Pensacola, FL)
Assignees: Nissan North America, Inc.; Florida Institute for Human & Machine Cognition, Inc.
G06Q50/30G05D1/0088G06Q10/047G06Q10/063G06Q10/20G07C5/008G07C5/0825G08G1/0112G08G1/052G06N20/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,839,473
App. No.
16/349,486
Granted
Nov 17, 2020
Kind
B2
Abstract

Methods, apparatuses, systems, and non-transitory computer readable storage media for monitoring vehicles including autonomous vehicles are described. The disclosed technology includes a vehicle monitoring system that receives vehicle data and external data associated with a vehicle and a corresponding predetermined area. The vehicle data includes a vehicle state of the vehicle and the external data includes external states of external objects. An issue type of the vehicle is determined based on the vehicle state and at least one of the external states. An indication of the issue type is generated for display on an interface.

Claims (61)

1. A method for vehicle monitoring, the method comprising:

receiving vehicle data and external data associated with a vehicle and a corresponding predetermined area, wherein the vehicle data includes a vehicle state of the vehicle and the external data includes external states of external objects;

determining an issue type of the vehicle based on the vehicle state and at least one of the external states; and

generating an indication of the issue type for display on an interface.

2. The method of claim 1 , further comprising:

generating route data based on location states and temporal states of the vehicle relative to expected location states and expected temporal states of the vehicle, wherein the issue type is based on the route data.

3. The method of claim 2 , further comprising:

generating expected destinations of the vehicle based on the expected location states and the expected temporal states;

grouping the external objects into clusters according to the expected destinations;

determining cluster sizes of each of the clusters; and

generating a congestion indication of each cluster that has a cluster size exceeding a threshold cluster size.

4. The method of claim 3 , further comprising:

modifying the expected temporal states of a portion of the external objects in the clusters that exceed the threshold cluster size, wherein the modifying comprises minimizing differences between the expected temporal states within each of the clusters.

5. The method of claim 1 , further comprising:

generating traffic data based on any of a number and a location of the external objects; and

determining a traffic issue based on the traffic data, wherein the indication of the issue type includes the traffic issue.

6. The method of claim 1 , further comprising:

sending a state query message to the vehicle, wherein the state query message includes a request for the vehicle data; and

in response to a predetermined time period elapsing before receiving a response to the state query message, retrieving object sensor data of any of the external objects located within the predetermined area.

7. The method of claim 1 , further comprising:

determining a velocity of the vehicle based on the vehicle data; and

in response to determining that the vehicle is in an immobile state based on the velocity of the vehicle, determining that the issue type is an immobility issue of the vehicle, wherein the indication of the issue type includes the immobility issue.

8. The method of claim 1 , wherein the vehicle data is based on vehicle sensor data of at least one sensor in the vehicle and further comprising:

determining an occupant state value based on the vehicle sensor data; and

in response to the occupant state value satisfying an occupant state condition, generating an occupant state indication.

9. The method of claim 1 , wherein the external data further comprises any of a vehicle velocity, velocities of the external objects, a vehicle orientation, orientations of the external objects, an autonomous mode state for the vehicle, a vehicle operational state, a vehicle energy state, traffic signal data, road closure data, construction data, road condition data, emergency vehicle route data, and zoning data.

10. A vehicle monitoring apparatus comprising:

a non-transitory memory; and

a processor configured to execute instructions stored in the non-transitory memory to:

receive vehicle data and external data associated with a vehicle and a corresponding predetermined area, wherein the vehicle data includes a vehicle state of a vehicle and the external data includes external states of external objects;

determine an issue type of the vehicle based on the vehicle state and at least one of the external states; and

generate an indication of the issue type for display on an interface.

11. The apparatus of claim 10 , wherein the processor is further configured to execute instructions stored in the memory to:

generate route data based on location states and temporal states of the vehicle relative to expected location states and expected temporal states of the vehicle, wherein the issue type is based on the route data.

12. The apparatus of claim 11 , wherein the processor is further configured to execute instructions stored in the memory to:

generate expected destinations of the vehicle based on the expected location states and the expected temporal states;

group the external objects into clusters according to the expected destinations;

determine cluster sizes of each of the clusters; and

generate a congestion indication of each cluster that has a cluster size exceeding a threshold cluster size.

13. The apparatus of claim 10 , wherein the processor is further configured to execute instructions stored in the memory to:

generate traffic data based on any of a number and a location of the external objects; and

determine a traffic issue based on the traffic data, wherein the indication of the issue type includes the traffic issue.

14. The apparatus of claim 10 , wherein the external data comprises any of vehicle velocity, velocities of the external objects, a vehicle orientation, orientations of the external objects, an autonomous mode state for the vehicle, a vehicle operational state, a vehicle energy state, traffic signal data, road closure data, construction data, road condition data, emergency vehicle route data, and zoning data.

15. The apparatus of claim 10 , wherein the processor is further configured to execute instructions stored in the memory to:

sending a state query message to the vehicle, wherein the state query message includes a request for the vehicle data; and

in response to a predetermined time period elapsing before receiving a response to the state query message, retrieving object sensor data of any of the external objects located within the predetermined area.

16. A non-transitory computer-readable storage medium including program instructions executable by one or more processors that, when executed, cause the one or more processors to perform operations, the operations comprising:

receiving vehicle data and external data associated with a vehicle and a corresponding predetermined area, wherein the vehicle data includes a vehicle state of a vehicle and the external data includes an external states of external objects;

determining an issue type of the vehicle based on the vehicle state and at least one of the external states; and

generating an indication of the issue type for display on an interface.

17. The non-transitory computer-readable storage medium of claim 16 , the operations further comprising:

generating route data based on location states and temporal states of the vehicle relative to expected location states and expected temporal states of the vehicle, wherein the issue type is based on the route data.

18. The non-transitory computer-readable storage medium of claim 17 , the operations further comprising:

generating expected destinations of the vehicle based on the expected location states and the expected temporal states;

grouping the external objects into clusters according to the expected destinations;

determining cluster sizes of each of the clusters; and

generating a congestion indication of each cluster that has a cluster size exceeding a threshold cluster size.

19. The non-transitory computer-readable storage medium of claim 16 , the operations further comprising:

generate traffic data based on any of a number and a location of the external objects; and

determine a traffic issue based on the traffic data, wherein the indication of the issue type includes the traffic issue.

20. The non-transitory computer-readable storage medium of claim 16 , wherein the external data comprises any of vehicle velocity, velocities of the external objects, a vehicle orientation, orientations of the external objects, an autonomous mode state for the vehicle, a vehicle operational state, a vehicle energy state, traffic signal data, road closure data, construction data, road condition data, emergency vehicle route data, and zoning data.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 24, 2021
From: NISSAN NORTH AMERICA, INC.
To: NISSAN MOTOR CO., LTD.
Reel/Frame 055699/0432 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 13, 2019
From: BRADSHAW, JEFFREY; JOHNSON, MATTHEW; VIGNATI, MICHAEL; BUNCH, LAWRENCE
To: FLORIDA INSTITUTE FOR HUMAN & MACHINE COGNITION, INC.
Reel/Frame 049161/0690 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 13, 2019
From: PEDERSEN, LIAM; SIERHUIS, MAARTEN; UTZ, HANS
To: NISSAN NORTH AMERICA, INC.
Reel/Frame 049161/0852 →
Continuity (2)
Provisional Application 62428082 · Nov 30, 2016
Related Publication 20200193549A1 · Jun 18, 2020
Cited By (1)
US 12,384,410