Granted Patent B2

US 12,236,692 · App. 17/338,390 · Granted Feb 25, 2025

Inventors: Yitian Wu (Plano, TX); Fatih Murat Porikli (Plano, TX); Lei Yang (Plano, TX); Yuan Ma (Plano, TX)

Assignee: Shenzhen Yinwang Intelligent Technologies Co., Ltd.

G06V20/597G06T7/70G06V10/82G06V20/58G06T2207/20084

Abstract

Claims (55)

1. A computer-implemented method for monitoring driver attentiveness in a vehicle, comprising:

collecting vehicle data and scene information from the vehicle while traveling on a route;

capturing a gaze of a driver to track a gaze direction and a duration of the driver while driving the vehicle on the route;

generating a reference heat map from the scene information and the vehicle data using a convolutional neural network (CNN) that comprises one or more convolutional layers and at least one fully-connected layer, the reference heat map identifying one or more regions in the scene information requiring driver attention for safe driving;

generating a driver gaze heat map based on the gaze direction and the duration of the driver while driving the vehicle using the CNN, the driver gaze heat map identifying one or more zones in the scene information viewed by the driver during the duration;

encoding, by a first subnetwork of a deep learning network, the reference heat map to generate a feature vector R:

encoding, by a second subnetwork of the deep learning network, the driver gaze heat map to generate a feature vector G;

determining a level of driver distraction of the driver in the vehicle based on a distance between the feature vector R and the feature vector G;

merging the feature vector R and the feature vector G to form a driver attention heat map, wherein the deep learning network determines whether the level of driver distraction meets or exceeds a threshold attentiveness level based on the driver attention heat map and generates driver attention results; and

outputting a recommendation to the driver of the vehicle according to the driver attention results.

2. The computer-implemented method of claim 1 , wherein tracking the gaze direction and the duration comprises:

capturing, by one or more sensors, a sequence of images of the driver over the duration;

estimating the gaze direction of the driver based on the gaze extracted from the sequence of images; and

applying the gaze direction to overlay the scene information to form the driver gaze heat map.

3. The computer-implemented method of claim 1 , wherein collecting the vehicle data includes recording a position of the vehicle during driving on the route and retrieving historical vehicle data from a database that is representative of the vehicle data over a predetermined period, and wherein collecting the scene information includes obtaining street images in real-time from one or more sensors or an online map.

4. The computer-implemented method of claim 1 , wherein the vehicle data includes at least one of vehicle speed data, turn indication data, light condition data, brake usage data, steering wheel usage data, global positioning system (GPS) data, gas pedal usage data, and navigation route data; and wherein the scene information includes at least one of scene images, online street maps, radar maps, weather, traffic and road conditions.

5. The computer-implemented method of claim 1 , wherein, when the threshold attentiveness level is met or exceeded, the recommendation is a warning to focus the gaze direction of the driver on one or more regions of the driver gaze heat map based on values associated with each of the one or more regions, where the values represent a level of attention required by the driver for each of the one or more regions.

6. The computer-implemented method of claim 1 , wherein one or more processors are located in the vehicle or communicatively coupled to the vehicle.

7. A system for monitoring driver attentiveness in a vehicle, comprising: one or more vehicle sensors configured to:

collect vehicle data and collect scene information from the vehicle while traveling on a route;

capture a gaze of a driver to track a gaze direction and a duration of the driver while driving the vehicle on the route; and

one or more processors configured to:

generate a reference heat map from the scene information and the vehicle data using a convolutional neural network (CNN) comprising one or more convolutional layers and at least one fully-connected layer, the reference heat map identifying one or more regions in the scene information requiring driver attention for safe driving;

generate a driver gaze heat map based on the gaze direction and the duration of the driver while driving the vehicle using the CNN, the driver gaze heat map identifying one or more zones in the scene information viewed by the driver during the duration;

encode, by a first subnetwork of a deep learning network, the reference heat map to generate a feature vector R;

encode, by a second subnetwork of the deep learning network, the driver gaze heat map to generate a feature vector G;

determine a level of driver distraction of the driver in the vehicle is based on a distance between the feature vector R and the feature vector G;

merge the feature vector R and the feature vector G to form a driver attention heat map, wherein the deep learning network determines whether the level of driver distraction meets or exceeds a threshold attentiveness level based on the driver attention heat map and generates driver attention results; and

output a recommendation to the driver of the vehicle according to the driver attention results.

8. The system of claim 7 , wherein tracking the gaze direction and the duration comprises:

capturing, by the one or more vehicle sensors, a sequence of images of the driver over the duration;

estimating the gaze direction of the driver based on the gaze extracted from the sequence of images; and

applying the gaze direction to overlay the scene information to form the driver gaze heat map.

9. The system of claim 7 , wherein collecting the vehicle data includes recording a position of the vehicle during driving on the route and retrieving historical vehicle data from a database that is representative of the vehicle data over a predetermined period, and wherein collecting the scene information includes obtaining street images in real-time from the one or more vehicle sensors or from an online map.

10. The system of claim 7 , wherein the vehicle data includes at least one of vehicle speed data, turn indication data, light condition data, brake usage data, steering wheel usage data, global positioning system (GPS) data, gas pedal usage data, and navigation route data, and wherein the scene information includes at least one of scene images, online street maps, radar maps, weather, traffic and road conditions.

11. The system of claim 7 , wherein, when the threshold attentiveness level is met or exceeded, the recommendation is a warning to focus the gaze direction of the driver on one or more regions of the driver gaze heat map based on values associated with each of the one or more regions, wherein the values represent a level of attention required by the driver for each of the one or more regions.

12. The system of claim 7 , wherein the one or more processors are located in the vehicle or communicatively coupled to the vehicle.

13. A non-transitory computer-readable medium storing computer instructions for monitoring driver attentiveness in a vehicle, that when executed by one or more processors, cause the one or more processors to perform the steps of:

collecting vehicle data and scene information from the vehicle while traveling on a route;

capturing a gaze of a driver to track a gaze direction and a duration of the driver while driving the vehicle on the route;

generating a reference heat map from the scene information and the vehicle data using a convolutional neural network (CNN) that comprises one or more convolutional layers and at least one fully-connected layer, the reference heat map identifying one or more regions in the scene information requiring driver attention for safe driving;

generating a driver gaze heat map based on the gaze direction and the duration of the driver while driving the vehicle using the CNN, the driver gaze heat map identifying one or more zones in the scene information viewed by the driver during the duration;

encoding, by a first subnetwork of a deep learning network, the reference heat map to generate a feature vector R:

encoding, by a second subnetwork of the deep learning network, the driver gaze heat map to generate a feature vector G;

determine a level of driver distraction of the driver in the vehicle based on a distance between the feature vector R and the feature vector G;

merging the feature vector R and the feature vector G to form a driver attention heat map, wherein the deep learning network determines whether the level of driver distraction meets or exceeds a threshold attentiveness level based on the driver attention heat map and generates driver attention results; and

outputting a recommendation to the driver of the vehicle according to the driver attention results.

14. The non-transitory computer-readable medium of claim 13 , wherein tracking the gaze direction and the duration comprises:

capturing, by one or more sensors, a sequence of images of the driver over the duration;

estimating the gaze direction of the driver based on the gaze extracted from the sequence of images; and

applying the gaze direction to overlay the scene information to form the driver gaze heat map.

15. The non-transitory computer-readable medium of claim 13 , wherein collecting the vehicle data includes recording a position of the vehicle during driving on the route and retrieving historical vehicle data from a database that is representative of the vehicle data over a predetermined period, and wherein collecting the scene information includes obtaining street images in real-time from one or more sensors.

16. The non-transitory computer-readable medium of claim 13 , wherein the vehicle data includes at least one of vehicle speed data, turn indication data, light condition data, brake usage data, steering wheel usage data, global positioning system (GPS) data, gas pedal usage data, and navigation route data; and wherein the scene information includes at least one of scene images, online street maps, radar maps, weather, traffic and road conditions.

17. The non-transitory computer-readable medium of claim 13 , wherein, when the threshold attentiveness level is met or exceeded, the recommendation is a warning to focus the gaze direction of the driver on one or more regions of the driver gaze heat map based on values associated with each of the one or more regions, wherein the values represent a level of attention required by the driver for each of the one or more regions.

18. The non-transitory computer-readable medium of claim 13 , wherein the one or more processors are located in the vehicle or communicatively coupled to the vehicle.

Assignments (2)

Continuity (2)

References Cited (13)