IP Library › Granted Patent US 12,012,107
Granted Patent B2
US 12,012,107 · App. 16/841,891 · Granted Jun 18, 2024

Operator monitoring and engagement

Inventors: Juan Eugene Gilbert (Gainesville, FL); Jerone N. Dunbar (Dublin, OH)
Assignee: University of Florida Research Foundation, Incorporated
B60W30/192B60W40/08B60W50/14G06N5/04G06N20/00G08B21/06B60W2040/0818B60W2540/229
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,012,107
App. No.
16/841,891
Granted
Jun 18, 2024
Kind
B2
Abstract

Methods, computing entities, systems, and computer program products for monitoring alertness of an operator of a vehicle are provided. In an example embodiment, a mobile computing entity receives biometric data and behavior data for the operator. The biometric data and behavior data captured by at least one of a wearable sensor, a visual sensor, or a vehicle sensor onboard the vehicle. The mobile computing entity determines an alertness level of the operator based at least in part on the biometric data and the behavior data using a model that is personalized for the operator. Responsive to determining that the alertness level of the operator satisfies a first treatment level threshold, the mobile computing entity initiates an operator engagement interaction to increase the alertness level of the operator.

Claims (73)

1. A method for monitoring alertness of an operator of a vehicle, the method comprising:

receiving, by a processor, biometric data and behavior data for the operator, the biometric data and the behavior data captured by at least one of a wearable sensor, a visual sensor, or a vehicle sensor onboard the vehicle;

determining, by the processor, an alertness level of the operator based at least in part on the biometric data and the behavior data using a model that is personalized for the operator;

responsive to determining that the alertness level of the operator satisfies a first treatment level threshold, performing, by the processor, an operator engagement interaction to increase the alertness level of the operator, wherein performing the operator engagement interaction comprises:

causing an elementary level arithmetic question to be audibly provided to the operator,

receiving an audible response originating from the operator corresponding to the elementary level arithmetic question,

in response to determining that the audible response included a correct answer to the elementary level arithmetic question, resetting, by the processor, an incorrect answer counter,

in response to determining that the audible response included an incorrect answer to the elementary level arithmetic question, iterating the incorrect answer counter,

in response to determining that the incorrect answer counter has reached a first threshold count, causing a suggestion that the operator pull over to be audibly provided to the operator, and

in response to determining that the incorrect answer counter has reached a second threshold count, determining that the alertness level of the operator satisfies a second treatment level threshold; and

responsive to determining that the alertness level of the operator satisfies the second treatment level threshold:

determining a current location of the vehicle;

identifying a break location based at least in part on the current location of the vehicle, the break location being a location where the vehicle is operable in an automated operation mode while the operator is able to take a break from operating the vehicle; and

causing route guidance to the break location to be provided.

2. The method of claim 1 , wherein the model is trained using a machine learning technique.

3. The method of claim 1 , wherein the operator engagement interaction is a voice-based interaction between a mobile computing entity and the operator.

4. The method of claim 3 , wherein the mobile computing entity comprises and/or is in communication with a speaker and a microphone.

5. The method of claim 1 , wherein the elementary level arithmetic question is addition and/or subtraction problems of two single digit numbers.

6. The method of claim 1 , wherein the biometric data comprises EEG data.

7. The method of claim 1 , further comprising:

determining whether the alertness level of the operator has improved as a result of the operator engagement interaction; and

responsive to determining that the alertness level of the operator has improved as a result of the operator engagement interaction, ceasing the operator engagement interaction.

8. The method of claim 1 , further comprising, responsive to determining that the alertness level of the operator satisfies the second treatment level threshold:

identifying one or more contacts and corresponding contact data based at least in part on an operator profile corresponding to the operator, and

notifying the one or more contacts.

9. The method of claim 8 , wherein notifying the one or more contacts comprises providing a notification to an electronic address corresponding to the one or more contacts.

10. The method of claim 1 , further comprising, responsive to determining that the alertness level of the operator satisfies the second treatment level threshold:

identifying a local emergency services department based at least in part on the current location; and

notifying the local emergency services department.

11. The method of claim 8 , wherein notifying the one or more contacts comprises initiating a phone call with a contact.

12. The method of claim 1 , further comprising:

after initiation of the operator engagement interaction, re-determining the alertness level of the operator; and

responsive to determining that the alertness level of the operator satisfies a second treatment level threshold, providing the operator with instructions for ceasing and/or pausing the operator's operation of the vehicle.

13. The method of claim 1 , further comprising:

determining whether the operator is responsive and coherent; and

responsive to determining that the operator is not responsive and/or not coherent,

initiating an automated shut down of the vehicle.

14. A mobile computing entity comprising at least one processor, a communications interface configured for communicating via at least one network, and at least one memory storing computer program code, the mobile computing entity being onboard a vehicle comprising one or more vehicle sensors, the at least one memory and the computer program code configured to, with the at least one processor, cause the mobile computing entity to at least:

receive biometric data and behavior data for an operator of the vehicle, the biometric data and the behavior data captured by at least one of a wearable sensor, a visual sensor, or a vehicle sensor of the one or more vehicle sensors;

determine an alertness level of the operator based at least in part on the biometric data and the behavior data using a model that is personalized for the operator;

responsive to determining that the alertness level of the operator satisfies a first treatment level threshold, perform an operator engagement interaction to increase the alertness level of the operator, wherein performing the operator engagement interaction comprises:

causing an elementary level arithmetic question to be audibly provided to the operator,

receiving an audible response originating from the operator corresponding to the elementary level arithmetic question,

in response to determining that the audible response included a correct answer to the elementary level arithmetic question, resetting, by the processor an incorrect answer counter,

in response to determining that the audible response included an incorrect answer to the elementary level arithmetic question, iterating the incorrect answer counter,

in response to determining that the incorrect answer counter has reached a first threshold count, causing a suggestion that the operator pull over to be audibly provided to the operator, and

in response to determining that the incorrect answer counter has reached a second threshold count, determining that the alertness level of the operator satisfies a second treatment level threshold; and

responsive to determining that the alertness level of the operator satisfies the second treatment level threshold:

determine a current location of the vehicle;

identify a break location based at least in part on the current location of the vehicle, the break location being a location where the vehicle is operable in an automated operation mode while the operator is able to take a break from operating the vehicle; and

cause route guidance to the break location to be provided.

15. The mobile computing entity of claim 14 , wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the mobile computing entity to at least:

after initiation of the operator engagement interaction, re-determine the alertness level of the operator; and

responsive to determining that the alertness level of the operator satisfies a second treatment level threshold, perform at least one of (a) notifying one or more contacts, the one or more contacts and corresponding contact data identified based at least in part on an operator profile corresponding to the operator or (b) providing the operator with instructions for ceasing and/or pausing the operator's operation of the vehicle.

16. The mobile computing entity of claim 14 , wherein the mobile computing entity comprises and/or is in communication with a speaker and a microphone, the operator engagement interaction is a voice-based interaction between the mobile computing entity and the operator, and the operator engagement interaction comprises the operator verbally responding to the elementary level arithmetic question.

17. A computer program product comprising at least one non-transitory computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising executable portions configured, when executed by a processor of an apparatus, to cause the apparatus to

receive biometric data and behavior data for an operator of a vehicle, the biometric data and the behavior data captured by at least one of a wearable sensor, a visual sensor, or a vehicle sensor onboard the vehicle;

determine an alertness level of the operator based at least in part on the biometric data and the behavior data using a model that is personalized for the operator;

responsive to determining that the alertness level of the operator satisfies a first treatment level threshold, perform an operator engagement interaction to increase the alertness level of the operator, wherein performing the operator engagement interaction comprises:

causing an elementary level arithmetic question to be audibly provided to the operator,

receiving an audible response originating from the operator corresponding to the elementary level arithmetic question,

in response to determining that the audible response included a correct answer to the elementary level arithmetic question, resetting, by the processor an incorrect answer counter,

in response to determining that the audible response included an incorrect answer to the elementary level arithmetic question, iterating the incorrect answer counter,

in response to determining that the incorrect answer counter has reached a first threshold count, causing a suggestion that the operator pull over to be audibly provided to the operator, and

in response to determining that the incorrect answer counter has reached a second threshold count, determining that the alertness level of the operator satisfies a second treatment level threshold; and

responsive to determining that the alertness level of the operator satisfies the second treatment level threshold:

determine a current location of the vehicle;

identify a break location based at least in part on the current location of the vehicle, the break location being a location where the vehicle is operable in an automated operation mode while the operator is able to take a break from operating the vehicle; and

cause route guidance to the break location to be provided.

18. The method of claim 1 ,

further comprising, responsive to determining that the vehicle is located at the location where the vehicle is operable in the automated operation mode, causing the vehicle to be operated in the automated operation mode.

19. The method of claim 18 , wherein operating the vehicle in the automated operation mode includes moving the vehicle from a travel lane of a road network and parking the vehicle.

20. The method of claim 1 , further comprising continuing to perform the operator engagement interaction as the operator operates the vehicle to the break location.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 7, 2020
From: GILBERT, JUAN EUGENE, DR.; DUNBAR, JERONE N., DR.
To: UNIVERSITY OF FLORIDA RESEARCH FOUNDATION, INCORPORATED
Reel/Frame 053134/0819 →
Continuity (2)
Provisional Application 62843643 · May 6, 2019
Related Publication 20200353933A1 · Nov 12, 2020