Granted Patent B2

US 12,499,696 · App. 18/348,566 · Granted Dec 16, 2025

Inventors: Alexander Barth (Wermelskirchen, DE); Timo Rehfeld (Cologne, DE)

Assignee: Aptiv Technologies AG

G06V20/597H04N23/667

Abstract

Claims (64)

1 . A computer implemented method for determining drowsiness of a driver in a vehicle, the method comprising:

monitoring the driver by capturing an image sequence of an interior of the vehicle via a camera system; and

determining, via a processing unit, a drowsiness signal indicative of the drowsiness of the driver by:

analyzing the image sequence to extract a characteristic pattern of the driver, the characteristic pattern being indicative of drowsiness of the driver;

determining an activity level of the driver based on at least one of the image sequence or an output signal provided by a sensor of the vehicle; and

selecting at least one of the extracted characteristic pattern or the activity level associated with the driver as a basis for determining the drowsiness signal, the selecting based on at least one of a frame rate of the image sequence, a confidence level assigned to the characteristic pattern, or a confidence level assigned to the activity level;

wherein determining the activity level of the driver from the output signal provided by the sensor of the vehicle comprises:

receiving a signal from a microphone in proximity to the driver; and

determining the activity level of the driver based on the signal of the microphone.

2 . The method according to claim 1 , wherein determining the activity level of the driver based on the image sequence comprises:

recognizing a movement pattern of the driver in the image sequence; and

deriving the activity level from the movement pattern.

3 . The method according to claim 2 , wherein the movement pattern is associated with a predefined facial region of the driver.

4 . The method according to claim 1 , wherein the characteristic pattern is associated with at least one of a predefined facial region or with a change of a body posture of the driver.

5 . The method according to claim 1 , wherein:

a drowsiness signal trend is derived from the image sequence; and

the drowsiness signal is determined by further considering the drowsiness signal trend.

6 . The method according to claim 1 , wherein the camera system is configured to operate in at least one of a high frame rate mode with a predetermined rate of image frames or a low frame rate mode with a reduced rate of image frames which is lower than the predetermined rate, and wherein:

if the camera system is operating in the high frame rate mode, the drowsiness signal is determined based on the extracted characteristic pattern; and

if the camera system is operating in the low frame rate mode, the drowsiness signal is determined based on the activity level.

7 . The method according to claim 6 , wherein if the camera system is operating in the low frame rate mode:

a base signal is determined based on the extracted characteristic pattern; and

the drowsiness signal is determined via a parametrizable function depending from the base signal and from the activity level.

8 . The method according to claim 7 , wherein:

the characteristic pattern is associated with a movement of eye lids of the driver; and

if the camera system is operating in the high frame rate mode, the drowsiness signal is determined based on at least one of an eye opening or an eye blink frequency which are derived from the movement of the driver's eye lids.

9 . The method according to claim 8 , wherein the camera system is operated in the low frame rate mode if the camera system is involved in an internal vehicle process which reduces the rate of image frames being available for monitoring the driver.

10 . The method according to claim 9 , wherein if the camera system is operated in the low frame rate mode and the processing unit fails in determining the drowsiness signal based on the activity level, the camera system is switched to the high frame rate mode and the internal vehicle process is deactivated.

11 . The method according to claim 10 , wherein:

the internal vehicle process includes a video call; and

the activity level is determined by monitoring an engagement of the driver during the video call.

12 . The method according to claim 11 , wherein:

a set of vehicle parameters is received by the processing unit; and

operating the camera system in the low frame rate mode is disabled if one of the set of vehicle parameters is greater than a predefined threshold.

13 . A computer system comprising:

one or more processors; and

memory storing one or more programs to be executed by the one or more processors, the one or more programs comprising instructions for:

monitoring a driver by capturing an image sequence of an interior of a vehicle via a camera system;

determining, via a processing unit, a drowsiness signal indicative of the drowsiness of the driver by:

analyzing the image sequence to extract a characteristic pattern of the driver, the characteristic pattern being indicative of drowsiness of the driver;

determining an activity level of the driver based on at least one of the image sequence or an output signal provided by a sensor of the vehicle; and

selecting at least one of the extracted characteristic pattern or the activity level associated with the driver as a basis for determining the drowsiness signal, and the selecting based on at least one of a frame rate of the image sequence, a confidence level assigned to the characteristic pattern, or a confidence level assigned to the activity level;

comparing the drowsiness signal with a predetermined threshold; and

in response to the drowsiness signal exceeding the predetermined threshold, performing at least one of: generating an alert within the vehicle to the driver; adapting an operation of an advanced driver assistance system (ADAS) to brake sooner; adapting an operation of the ADAS to keep a larger distance from other vehicles; or lowering a warning threshold of the ADAS.

14 . The computer system according to claim 13 , wherein determining the activity level of the driver based on the image sequence comprises:

recognizing a movement pattern of the driver in the image sequence; and

deriving the activity level from the movement pattern.

15 . The computer system according to claim 14 , wherein the movement pattern is associated with a predefined facial region of the driver.

16 . The computer system according to claim 13 , wherein determining the activity level of the driver from the output signal provided by the sensor of the vehicle comprises:

receiving a signal from a microphone in proximity to the driver; and

determining the activity level of the driver based on the signal of the microphone.

17 . A non-transitory computer readable medium storing one or more programs, the one or more programs comprising instructions, which, when executed by a computer system with one or more processors, cause the computer system to perform operations comprising:

monitoring a driver by capturing an image sequence of an interior of a vehicle via a camera system;

determining, via a processing unit, a drowsiness signal indicative of the drowsiness of the driver by:

analyzing the image sequence to extract a characteristic pattern of the driver, the characteristic pattern being indicative of drowsiness of the driver;

determining an activity level of the driver based on at least one of the image sequence or an output signal provided by a sensor of the vehicle; and

selecting at least one of the extracted characteristic pattern and the activity level associated with the driver as a basis for determining the drowsiness signal, and the selecting is based on at least one of a frame rate of the image sequence, a confidence level assigned to the characteristic pattern, or a confidence level assigned to the activity level;

comparing the drowsiness signal with a predetermined threshold; and

in response to the drowsiness signal exceeding the predetermined threshold, performing at least one of: generating an alert within the vehicle to the driver; adapting an operation of an advanced driver assistance system (ADAS) to brake sooner; adapting an operation of the ADAS to keep a larger distance from other vehicles; or

lowering a warning threshold of the ADAS.

18 . The non-transitory computer readable medium according to claim 17 , wherein determining the activity level of the driver based on the image sequence comprises:

recognizing a movement pattern of the driver in the image sequence; and

deriving the activity level from the movement pattern.

19 . The non-transitory computer readable medium according to claim 18 , wherein the movement pattern is associated with a predefined facial region of the driver.

Assignments (5)

Priority Claims (1)

Continuity (1)

References Cited (15)