Systems and methods for operator monitoring and fatigue detection
Disclosed are systems and methods for operator monitoring and fatigue detection. Disclosed systems and methods may be used to monitor operators in real-time in order to identify and/or prevent any possible cause for a potential hazard by alerting the operator and/or taking preventive measures in the event of receiving a failed response from the operator. In some embodiments, operators may be monitored by a wearable device including a plurality of sensors. In some embodiments, a system for operator monitoring may include an operator monitoring device configured to determine operator positional data, a vehicle base station configured to determine vehicle positional data, apply a machine learning based algorithm to determine if the operator is in a state of reduced alertness, and perform a corrective measure responsive to determining that the operator is in a state of reduced alertness, and a server system configured to train the machine learning based algorithm.
1 . A system for operator monitoring comprising:
an earbud comprising an operator positional information sensor and a biosensor configured to record operator positional information and operator biosignals from an operator; and
a vehicle base station communicatively coupled to the earbud, the vehicle base station comprising one or more vehicle positional sensors and a user status artificial intelligence module configured to:
establish a zero-axis for the operator within a three-dimensional space of a vehicle by analyzing the operator positional information and the operator biosignals during a guided training process comprising a range of one or more gestures and positions of the operator;
determine operator alertness status by analyzing the operator positional information, the operator biosignals, and vehicle positional information from the vehicle positional sensors during operation of the vehicle to determine at least one of:
whether the operator is exhibiting electrical activity indicative of eye blinking via the biosensor,
whether the operator is exhibiting central nervous system electrical activity indicative of lack of attentiveness or fatigue,
whether the operator is exhibiting a change in a line of sight from the zero-axis to a new axis for at least a threshold period of time via the operator positional information sensor,
whether the operator is exhibiting a change in a line of sight from the normal zero-axis to the new axis with at least a threshold frequency via the operator positional information sensor, or
whether a head position of the operator is deviating from a standard position via the operator positional information sensor,
initiate a first corrective measure responsive to determining that the operator is in a state of reduced alertness,
monitor the operator positional information and operator biosignals for an indication that the operator acknowledged the first corrective measure via a predetermined gesture, and
upon determining that the indication is not received within a predetermined time period, initiate a second corrective measure.
2 . The system of claim 1 , wherein the biosensor comprises an electroencephalogram (EEG) sensor, an electrocardiogram (ECG) sensor, or an electromyography sensor (EMG).
3 . The system of claim 1 , wherein the user status artificial intelligence module comprises:
a learning sub-module comprising normal and abnormal operator gesture patterns for training a random forest algorithm; and
a prediction sub-module configured to apply the trained random forest algorithm to the operator biosignals and the operator positional information received from the earbud.
4 . The system of claim 1 , wherein the vehicle positional sensors comprise an accelerometer, gyroscope, magnetometer, or global positioning system (GPS).
5 . The system of claim 1 , wherein the operator positional sensors comprise an accelerometer, gyroscope or magnetometer.
6 . The system of claim 1 , wherein the earbud comprises a pressure sensor.
7 . The system of claim 1 , wherein the at least a portion of the earbud is positioned within an ear canal of an operator.
8 . The system of claim 1 , wherein the at least a portion of the earbud is positioned adjacent to a back side of a concha.
9 . The system of claim 1 , wherein the system comprises:
a server communicatively coupled to the earbud or the vehicle base station, wherein the server comprises an asset tracking module configured to receive and display vehicle positional data from a plurality of vehicles.
10 . The system of claim 1 , wherein the first corrective measure comprises at least one of an audio alarm and a visual alarm, and the second corrective measure comprises at least one of an audio alarm, a visual alarm, and altered operation of a vehicle on which the vehicle base station is located, wherein the predetermined gesture comprises one or more of a jaw clench and an eye blink.
11 . A method for operator monitoring comprising:
recording, via an earbud comprising an operator positional information sensor and a biosensor, operator positional information via the operator positional information sensor and operator biosignals via the biosensor from an operator;
transmitting the operator positional information and the operator biosignals to a vehicle base station communicatively coupled to the earbud, wherein the vehicle base station is associated with a vehicle;
generating, at the vehicle base station, vehicle positional information;
establishing, at the vehicle base station, a zero-axis for the operator within a three-dimensional space of the vehicle by applying a user stats artificial intelligence module to the operator positional information and the operator biosignals while performing a guided training process comprising a range of one or more gestures and positions of the operator;
determining, at the vehicle base station, operator alertness status by applying the user status artificial intelligence module to the operator positional information, the operator biosignals, and the vehicle positional information during operation of the vehicle by determining at least one of:
whether the operator is exhibiting electrical activity indicative of eye blinking via the biosensor,
whether the operator is exhibiting central nervous system electrical activity indicative of lack of attentiveness or fatigue,
whether the operator is exhibiting a change in a line of sight from the zero-axis to a new axis for at least a threshold period of time via the operator positional information sensor,
whether the operator is exhibiting a change in a line of sight from the zero-axis to the new axis with at least a threshold frequency via the operator positional information sensor, or
whether a head position of the operator is deviating from a standard position via the operator positional information sensor;
initiating, at the vehicle base station, a first corrective measure responsive to determining that the operator is in a state of reduced alertness;
monitoring, at the vehicle base station, the operator positional information and operator biosignals for an indication that the operator acknowledged the first corrective measure via a predetermined gesture; and
upon determining that the indication is not received within a predetermined time period, initiating, at the vehicle base station, a second corrective measure.
12 . The method of claim 11 , wherein recording the operator biosignals comprises:
recording at least one of electroencephalogram (EEG) signals, electrocardiogram (ECG) signals, and electromyography (EMG) signals;
identifying a most relevant signal by applying a filter to the recorded signals; and
applying feature extraction to the identified most relevant signal.
13 . The method of claim 11 , wherein the operator positional information sensor comprises at least one of an accelerometer, gyroscope, or magnetometer.
14 . The method of claim 11 , the method comprising:
training the user status artificial intelligence module by providing normal and abnormal operator gesture patterns to a random forest algorithm.
15 . The method of claim 11 , wherein the user status artificial intelligence module comprises a random forest algorithm.
16 . The method of claim 11 , wherein the first corrective measure comprises at least one of an audio alarm and a visual alarm
and the second corrective measure comprises at least one of an audio alarm, a visual alarm, and altering operation of the vehicle on which the vehicle base station is located.
17 . The method of claim 11 , wherein the predetermined gesture is comprises one or more of a jaw clench and an eye blink.
18 . The method of claim 11 , the method comprising:
transmitting at least one of the operator positional information and the vehicle position information to a server system having an asset tracking module.
19 . The method of claim 11 , the method comprising:
positioning the at least a portion of the earbud within an ear canal of an operator or adjacent to a back side of a concha.
20 . The method of claim 11 , wherein generating the vehicle positional information comprises determining a vehicle position based on at least one of an accelerometer, gyroscope, magnetometer and global positioning device of the vehicle base station.