IP Library Granted Patent US 11,281,944
Granted Patent B2
US 11,281,944 · App. 16/430,810 · Granted Mar 22, 2022

System and method for contextualized vehicle operation determination

Inventors: Suchitra Sathyanarayana (Palo Alto, CA); Ravi Kumar Satzoda (Palo Alto, CA); Alex Thompson (Palo Alto, CA); Michael Gleeson-May (Palo Alto, CA); Ludmila Levkova (Palo Alto, CA)
Assignee: Nauto, Inc.
G06K9/6289B60W30/00B60W30/095B60W40/08G05D1/021G05D1/0214G05D1/0221G05D1/0251G06K9/00805G06K9/00845G06K9/6257G06K9/6267G06K9/6293G06N3/04G06N3/084G06N20/00G06Q10/00G06T7/50G06T7/70G07C5/0866G08G1/0112G08G1/0129G08G1/0145G08G1/04G08G1/166B60W2040/0872B60W2540/22B60W2554/00G05D2201/0213G06Q10/20G06T2207/20076H04N5/247
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,281,944
App. No.
16/430,810
Granted
Mar 22, 2022
Kind
B2
Abstract

A method for determining event data including: sampling a first data stream within a first time window at a first sensor of an onboard vehicle system coupled to a vehicle, extracting interior activity data from the first data stream; determining an interior event based on the interior activity data; sampling a second data stream within a second time window at a second sensor of the onboard vehicle system; extracting exterior activity data from the second image stream; determining an exterior event based on the exterior activity data; correlating the exterior event and the interior event to generate combined event data; automatically classifying the combined event data to generate an event label; and automatically labeling the first time window of the first data stream and the second time window of the second data stream with the combined event label to generate labeled event data.

Claims (49)

1. A method involving a first image stream and a second image stream, the method comprising:

sampling the first image stream within a first time window at an interior-facing camera of an onboard vehicle system, wherein the onboard vehicle system is integrated into a mountable unit configured to couple to a vehicle;

electronically extracting interior activity data indicating a state of a driver from the first image stream;

determining an interior event based on the interior activity data;

sampling the second image stream within a second time window at an exterior-facing camera of the onboard vehicle system, wherein the first time window and the second time window are coextensive;

electronically extracting exterior activity data from the second image stream;

determining an exterior event based on the exterior activity data;

correlating the interior event and the exterior event to generate combined event data, wherein the act of correlating comprises:

determining a weight based on the driver distraction event,

determining a metric associated with the exterior event, and

weighting the metric associated with the exterior event by the weight to obtain a weighted metric; and

classifying the combined event data

wherein the act of determining the interior event, the act of determining the exterior event, and the act of correlating the interior event and the exterior event to generate combined event data, are performed by one or more electronic processing units.

2. The method of claim 1 , wherein the act of electronically extracting the exterior activity data comprises:

determining a relative position between the vehicle and an object depicted in the second image stream.

3. The method of claim 2 , wherein the object is a secondary vehicle, and wherein the act of determining the exterior event comprises determining that the relative position between the vehicle and the secondary vehicle is below a threshold distance.

4. The method of claim 1 , wherein the exterior event comprises a near-miss event, wherein the near-miss event occurs within the first-time window, and wherein the act of correlating the interior event and the exterior event comprises associating the interior event with the near-miss event; and

wherein the combined event data comprises the near-miss event, and the associated interior event.

5. The method of claim 1 , wherein the state of the driver comprises a distracted state, and the electronically extracted interior activity data indicates the distracted state of the driver.

6. The method of claim 1 , wherein the metric associated with the exterior event indicates a severity of the exterior event.

7. The method of claim 1 , further comprising generating training data based on the classified combined event data.

8. The method of claim 7 , further comprising:

training a vehicle control model using the training data; and

transmitting a copy of the vehicle control model to the vehicle for controlling the vehicle.

9. A method involving a first image stream and a second image stream, the method comprising:

sampling the first data stream within a first time window at a first sensor of an onboard vehicle system that is configured to couple to a vehicle;

electronically extracting interior activity data from the first data stream;

determining an interior event based on the interior activity data;

sampling the second data stream within a second time window at a second sensor of the onboard vehicle system;

electronically extracting exterior activity data from the second image stream;

determining an exterior event based on the exterior activity data;

correlating the exterior event and the interior event to generate combined event data, wherein the act of correlating the interior event with the exterior event comprises calculating a probability that the interior event caused the exterior event, wherein the combined event data comprises the exterior event, the interior event, and the probability; and

classifying the combined event data

wherein the act of determining the interior event, the act of determining the exterior event, and the act of correlating the exterior event and the interior event to generate combined event data, are performed by one or more electronic processing units.

10. The method of claim 9 , wherein the first data stream comprises gyroscope data.

11. The method of claim 10 , wherein the act of electronically extracting the interior activity data comprises determining a mean time interval between steering inputs based on the gyroscope data and

wherein the act of determining the interior event comprises determining a driver distraction event based on the mean time interval exceeding a threshold duration.

12. The method of claim 9 , wherein the act of correlating the exterior event and the interior event comprises:

determining a weight based on the interior event;

determining a metric associated with the exterior event; and

weighting the metric associated with the exterior event by the weight to obtain a weighted metric.

13. The method of claim 9 , wherein the onboard vehicle system is integrated into a mountable unit configured to couple to the vehicle.

14. The method of claim 9 , wherein the interior event and the exterior event are correlated based on relative geometric arrangement between the first sensor and the second sensor.

15. The method of claim 9 , wherein the first data stream comprises a first image stream, and the first sensor comprises a first camera of the onboard vehicle system; and

wherein the second data stream comprises a second image stream, and the second sensor comprises a second camera of the onboard vehicle system.

16. The method of claim 9 , wherein the first time window is coextensive with the second time window.

17. The method of claim 9 , further comprising generating training data based on the classified combined event data.

18. The method of claim 17 , further comprising:

training a vehicle control model using the training data.

Assignments (5)
RELEASE OF SECURITY INTEREST Recorded Jun 19, 2026
From: ORIX GROWTH CAPITAL, LLC
To: NAUTO, INC.
Reel/Frame 075016/0824 →
SECURITY INTEREST Recorded Aug 8, 2025
From: NAUTO, INC.
To: ORIX GROWTH CAPITAL, LLC, AS AGENT
Reel/Frame 071976/0818 →
SECURITY INTEREST Recorded Nov 10, 2022
From: NAUTO, INC.
To: SILICON VALLEY BANK
Reel/Frame 061722/0392 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 4, 2019
From: SATHYANARAYANA, SUCHITRA; SATZODA, RAVI KUMAR; THOMPSON, ALEX; GLEESON-MAY, MICHAEL; LEVKOVA, LUDMILA
To: NAUTO GLOBAL LIMITED
Reel/Frame 049360/0717 →
CHANGE OF NAME Recorded Jun 4, 2019
From: NAUTO GLOBAL LIMITED
To: NAUTO, INC.
Reel/Frame 049360/0814 →
Continuity (3)
Continuation 16011437 · Jun 18, 2018
Provisional Application 62521091 · Jun 16, 2017
Related Publication 20190286948A1 · Sep 19, 2019