Driver monitoring with real time feedback
An apparatus comprising an interface and a processor. The interface may be configured to receive pixel data of a driver of a vehicle and an environment near the vehicle. The processor may be configured to process the pixel data arranged as video frames, generate a text description of the video frames, store the text description and a timestamp as metadata with the video frames, store triggers, compare the triggers to the text description, and generate a driver score in response to the comparison. A first AI model may perform video to text analysis to generate the text description. A second AI model may be configured to perform the comparison and generate the driver score in response to the triggers. The plurality of triggers may comprise a plain text description of driver behavior and an operation of the vehicle in the environment.
1 . An apparatus comprising:
an interface configured to receive pixel data of (i) a driver of a vehicle and (ii) an environment near said vehicle; and
a processor configured to (i) process said pixel data arranged as video frames, (ii) generate a text description of said video frames, (iii) store said text description and a timestamp corresponding to said text description as metadata with said video frames, (iv) store a plurality of, (v) perform a comparison of said triggers to said text description of said video frames, and (vi) generate a driver score in response to said comparison of said triggers to said text description, wherein
(i) a first AI model is configured to perform video to text analysis of said video frames to generate said text description of said driver and said environment,
(ii) a second AI model is configured to (a) perform said comparison and (b) generate said driver score in response to said triggers, and
(iii) said plurality of triggers comprise a plain text description of (a) driver behavior and (b) an operation of said vehicle in said environment.
2 . The apparatus according to claim 1 , further comprising a sensor fusion module, wherein
(i) said sensor fusion module is configured to read data from (a) a CAN bus of said vehicle and (b) one or more vehicle sensors,
(ii) said sensor fusion module is configured to (a) make inferences in response to said data and (b) generate a data timestamp associated with said inferences,
(iii) said first AI model is configured to generate a second text description in response to said inferences, and
(iv) said comparison by said second AI model is further configured to compare said second text description at said data timestamp and said text description of said video frames at said timestamp with said triggers.
3 . The apparatus according to claim 2 , wherein said vehicle sensors comprise one or more of a gyroscope, an accelerometer, a radar system, and a lidar system.
4 . The apparatus according to claim 1 , wherein said driver score is configured to provide real time feedback for driving improvement to said driver.
5 . The apparatus according to claim 1 , wherein said driver score further comprises a notification of a detection of one or more of said triggers.
6 . The apparatus according to claim 5 , wherein said notification is communicated to an owner of said vehicle.
7 . The apparatus according to claim 6 , wherein (i) said vehicle is one of a plurality of vehicles in a vehicle fleet and (ii) said owner of said vehicle fleet is a fleet manager.
8 . The apparatus according to claim 1 , further comprising a large language model (LLM) AI model configured to receive input from a user device, wherein (i) said input comprises a natural language query about said text description of said video frames and (ii) said LLM AI model is configured to parse said natural language query, compare said natural language query to said text description of said video frames and generate a natural language response to said natural language query.
9 . The apparatus according to claim 8 , wherein (i) said input further comprises an input trigger provided using natural language and (ii) said LLM AI model is configured to parse said input trigger and add said input trigger to said plurality of triggers.
10 . The apparatus according to claim 1 , wherein (i) said driver score is uploaded to a remote database, (ii) said remote database is configured to store a plurality of driver scores comprising said driver score from said apparatus and additional driver scores from a vehicle fleet, and (iii) a large language model (LLM) AI model is configured to (a) access said plurality of driver scores in said remote database, (b) receive a natural language query from a manager of said vehicle fleet, (c) compare said natural language query to said plurality of driver scores and (d) generate a natural language response to said natural language query.
11 . The apparatus according to claim 10 , wherein (i) said natural language query comprises an alert threshold, (ii) said driver scores are tracked for said alert threshold and (iii) said LLM AI model is configured to generate said natural language response comprising a notification about said alert threshold in response to monitoring said driver scores for said alert threshold.
12 . The apparatus according to claim 11 , wherein said alert threshold comprises at least one of (i) a particular amount for said driver score and (ii) a particular one of said triggers.
13 . The apparatus according to claim 1 , wherein said plurality of triggers correspond to characteristics about said driver of said vehicle.
14 . The apparatus according to claim 13 , wherein said characteristics about said driver of said vehicle comprise one or more of driver distraction and driver drowsiness.
15 . The apparatus according to claim 1 , wherein said plurality of triggers correspond to behavior of said driver of said vehicle.
16 . The apparatus according to claim 15 , wherein said behavior of said driver comprises one or more of rapid acceleration, lane cutting, tailgating, rapid braking, indicator use, speeding, and road rule violations.
17 . The apparatus according to claim 1 , wherein said plurality of triggers correspond to said environment of said vehicle.
18 . The apparatus according to claim 17 , wherein said environment of said vehicle comprises one or more of a cargo temperature, information from telematics, time of day, and weather.
19 . The apparatus according to claim 1 , wherein (i) said driver score comprises a total number of points, and (ii) an adjustment to said driver score comprises a pre-defined deduction of points corresponding to a particular one of said triggers detected from said total number of points.
20 . The apparatus according to claim 1 , wherein (i) said driver score comprises a probability of an incident, and (ii) said of said incident comprises (a) determining a total distance of a route, (b) determining an expected number of infractions of said triggers for said route based on said total distance and (c) comparing said expected number of infractions to a number of said triggers detected.