IP Library Granted Patent US 11,990,036
Granted Patent B2
US 11,990,036 · App. 17/330,635 · Granted May 21, 2024

Driver behavior monitoring

Inventors: David Jonathan Julian (San Diego, CA); Avneesh Agrawal (Bangaluru, IN)
Assignee: Netradyne, Inc.
G08G1/04G06F18/24133G06V10/40G06V10/764G06V10/82G06V20/56G06V20/58G06V20/597G07C5/0808G07C5/0866G08G1/0112G08G1/0133B60W40/09B60W2420/403G06V20/44
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,990,036
App. No.
17/330,635
Granted
May 21, 2024
Kind
B2
Abstract

Systems and methods provide, implement, and use using a computer-vision based methods of context-sensitive monitoring and characterization of driver behavior. Additional systems and methods are provided for unsupervised learning of action values, monitoring of a driver's environment, and transmitting visual information from a client to a server.

Claims (40)

1. A method comprising:

receiving, by at least one processor of a computing device, an image captured by a camera mounted on or in a vehicle;

detecting, by the at least one processor, an object in an image from the camera using a neural network trained to output a bounding box surrounding the object in the image and a class identification data for the object;

determining, by the at least one processor, whether the object applies to the vehicle;

determining, by the at least one processor, an event based on movement of the vehicle in the presence of the object when the object applies to the vehicle; and

determining, by the at least one processor, an extent to which a driver of the vehicle performs a particular action associated with the event.

2. The method according to claim 1 , wherein classifying comprises determining a probability that the object belongs to a class.

3. The method according to claim 1 , wherein the output comprises a distance of the object from a specified point on the vehicle.

4. The method according to claim 3 , wherein the specified point is a front bumper of the vehicle.

5. The method according to claim 1 , wherein the class identification data comprises pixel values corresponding to an inside of the bounding box.

6. The method according to claim 1 , wherein the image comprises a cropped image for the detected object.

7. The method according to claim 1 , further comprising increasing, by the at least one processor, confidence associated with detecting the object in a single frame upon detecting the object in a plurality of frames.

8. The method according to claim 1 , further comprising decreasing, by the at least one processor, confidence associated with detecting the object in a single frame when the object is not detected in a consecutive frame.

9. The method according to claim 1 , wherein determining the extent comprises determining a responsibility of the driver using a machine learning model that outputs a fault of the driver.

10. The method according to claim 9 , further comprising inputting, by the at least one processor, a plurality of feature vectors for a plurality of frames into the machine learning model.

11. The method according to claim 10 , wherein the plurality of feature vectors comprises at least one of a vehicle speed, a lane position, a relative position of another vehicle, or a relative position to a pedestrian.

12. The method according to claim 9 , wherein the machine learning model is a deep neural network or a support vector machine.

13. The method according to claim 1 , wherein the event represents lane driving, tailgating, or obeying a traffic sign.

14. The method according to claim 1 , wherein the neural network is a fast region-based convolutional network.

15. The method according to claim 1 , wherein the neural network generates the bounding box.

16. A computer program product comprising a non-transitory computer-readable medium having instructions stored thereon, the instructions being executable by one or more processors configured to:

receive an image captured by a camera mounted on or in a vehicle;

detect an object in an image from the camera using a neural network trained to output a bounding box surrounding the object in the image and a class identification for the object;

determine whether the object applies to the vehicle;

determine an event based on movement of the vehicle in the presence of the object when the object applies to the vehicle; and

determine an extent to which a driver of the vehicle performs a particular action associated with the event.

17. The computer-program product according to claim 16 , wherein the one or more processors are configured to determine a probability that the object belongs to a class.

18. The computer-program product according to claim 16 , wherein the output comprises a distance of the object from a specified point on the vehicle.

19. The computer-program product according to claim 18 , wherein the specified point is a front bumper of the vehicle.

20. The computer-program product according to claim 16 , wherein the class identification data comprises pixel values corresponding to an inside of the bounding box.

21. The computer-program product according to claim 16 , wherein the image comprises a cropped image for the detected object.

22. The computer-program product according to claim 16 , wherein the one or more processors are configured to increase confidence associated with detecting the object in a single frame upon detecting the object in a plurality of frames.

23. The computer-program product according to claim 16 , wherein the one or more processors are configured to decrease confidence associated with detecting the object in a single frame when the object is not detected in a consecutive frame.

24. The computer-program product according to claim 16 , wherein the one or more processors are configured to determine a responsibility of the driver using a machine learning model that outputs a fault of the driver.

25. The computer-program product according to claim 24 , wherein the one or more processors are configured to input a plurality of feature vectors for a plurality of frames into the machine learning model.

26. The computer-program product according to claim 25 , wherein the plurality of feature vectors comprises at least one of a vehicle speed, a lane position, a relative position of another vehicle, or a relative position to a pedestrian.

27. The computer-program product according to claim 24 , wherein the machine learning model is a deep neural network or a support vector machine.

28. The computer-program product according to claim 16 , wherein the event represents lane driving, tailgating, or obeying a traffic sign.

29. The computer-program product according to claim 16 , wherein the neural network is a fast region-based convolutional network.

30. The computer-program product according to claim 16 , wherein the neural network generates the bounding box.

Assignments (3)
SECURITY INTEREST Recorded Apr 6, 2026
From: NETRADYNE, INC.
To: FIRST-CITIZENS BANK & TRUST COMPANY
Reel/Frame 075435/0670 →
SECURITY INTEREST Recorded Apr 6, 2026
From: NETRADYNE, INC.
To: FIRST-CITIZENS BANK & TRUST COMPANY
Reel/Frame 075359/0194 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 26, 2021
From: JULIAN, DAVID JONATHAN; AGRAWAL, AVNEESH
To: NETRADYNE, INC.
Reel/Frame 056355/0610 →
Continuity (6)
Continuation 16725763 · Dec 23, 2019
Continuation 16507894 · Jul 10, 2019
Continuation 15437646 · Feb 21, 2017
Continuation PCTUS2017013062 · Jan 11, 2017
Provisional Application 62277470 · Jan 11, 2016
Related Publication 20210280056A1 · Sep 9, 2021