Attention, comprehension, and drowsiness monitoring via head mounted device supporting augmented and mixed reality experiences
A head mounted display (HMD) system includes an HMD device worn on a head of a user. The HMD device incorporates electroencephalography (EEG) interfaces for monitoring the brain of a human subject during interaction with the HMD device. Fluctuations in electrical potential that are observed via the EEG interfaces may be used to detect event-related potentials (ERPs). The HMD system may programmatically perform one or more operations in response to detecting ERPs. The HMD system may further include off-board devices that communicate with the HMD device over a wireless communications network.
1. A computer-implemented method comprising:
presenting, by one or more processors of a machine, a visual or auditory stimulus to a user of a head mounted device (HMD) worn on a head of the user, the HMD being configured to display augmented reality content in a transparent display of the HMD;
obtaining, by one or more processors of the machine, a set of electroencephalography (EEG) signals via a set of EEG electrodes of the HMD, each EEG electrode being configured to detect fluctuations of an electrical potential at a corresponding location relative to the head of the user in response to the visual or auditory stimulus being presented;
computing, by one or more processors of the machine, a level of drowsiness of the user based on the set of EEG signals; and
determining, by one or more processors of the machine, the augmented reality content in response to a change in the level of drowsiness.
2. The computer-implemented method of claim 1 , further comprising:
generating a notification in response to the level of drowsiness transgressing a drowsiness threshold; and
presenting the notification and an indication of the level of drowsiness in the transparent display of the HMD.
3. The computer-implemented method of claim 1 , wherein the visual or auditory stimulus includes instructions, and the method further comprises:
computing a level of comprehension of the instructions by the user based on the set of EEG signals; and
adjusting the augmented reality content in response to the level of comprehension transgressing a comprehension threshold.
4. The computer-implemented method of claim 1 , wherein the visual or auditory stimulus includes instructions, and the method further comprises:
computing a level of attention of the user to the instructions based on the set of EEG signals; and
adjusting the augmented reality content in response to the level of attention transgressing an attention threshold.
5. The computer-implemented method of claim 1 , wherein the computing of the level of drowsiness comprises:
detecting an event-related potential (ERP) in the user's response to presentation of the visual or auditory stimulus by searching within the set of EEG signals for at least one of:
an ERP-P3b waveform having a positive deflection in electrical potential within an ERP-P3b time-based search window following the presenting of the visual or auditory stimulus, or
an ERP-N400 waveform having a negative deflection in electrical potential within an ERP-N400 time-based search window following the presenting of the visual or auditory stimulus; and
for each waveform present within the set of EEG signals, measuring a peak-amplitude value at a peak of that waveform and a peak-time value at the peak of that waveform relative to a time of presentation of the visual or auditory stimulus.
6. The computer-implemented method of claim 5 , further comprising:
communicating the level of drowsiness for the user with respect to the stimulus based on the measurements of the peak-amplitude and the peak-time value of the ERP-P3b waveform.
7. The computer-implemented method of claim 5 , further comprising:
communicating a level of attentiveness for the user with respect to the stimulus based on the measurements of the peak-amplitude and the peak-time value of the ERP-P3b waveform.
8. The computer-implemented method of claim 5 , further comprising:
communicating a level of comprehension for the user with respect to an information-bearing message of the stimulus based on the measurements of the peak-amplitude and the peak-time value of the ERP-N400 waveform.
9. The computer-implemented method of claim 1 , further comprising:
obtaining time-based imagery of an eye of the user via an ocular camera of the head mounted device; and
obtaining measurements of eye movement with respect to the eye of the user by processing the time-based imagery;
wherein the computing of the level of drowsiness for the human subject is further based on the measurements of eye movement.
10. The computer-implemented method of claim 1 , further comprising:
obtaining at least one of a Theta band, an Alpha band, or a Beta band; and
wherein the computing of the level of drowsiness for the user is further based on at least one of a measurement of amplitude or a regularity of frequency of the set of EEG signals within the one or more of the bands.
11. A head-mounted device (HMD) comprising:
a transparent display configured to display augmented reality content and present a visual stimulus to a user of the HMD worn on a head of the user;
a speaker configured to present an audio stimulus to the user of the HMD;
a set of EEG electrodes of the HMD, each EEG electrode configured to detect fluctuations of an electrical potential at a corresponding location relative to the head of the user of the HMD in response to the visual and auditory stimulus being presented;
a hardware processor configured to perform operations comprising:
obtaining a set of electroencephalography (EEG) signals via the set of EEG electrodes of the HMD;
computing a level of drowsiness of the user based on the EEG signals; and
adjusting the augmented reality content in response to a change in the level of drowsiness.
12. The HMD of claim 11 , wherein the operations further comprise:
generating a notification in response to the level of drowsiness transgressing a drowsiness threshold; and
presenting the notification and an indication of the level of drowsiness in the transparent display of the HMD.
13. The HMD of claim 11 , wherein the visual and auditory stimulus includes instructions, wherein the operations further comprise:
computing a level of comprehension of the instructions by the user based on the set of EEG signals; and
adjusting the augmented reality content in response to the level of comprehension transgressing a comprehension threshold.
14. The HMD of claim 11 , wherein the visual or auditory stimulus includes instructions, wherein the operations further comprise:
computing a level of attention of the user to the instructions based on the set of EEG signals; and
adjusting the augmented reality content in response to the level of attention transgressing an attention threshold.
15. The HMD of claim 11 , wherein computing the level of drowsiness comprises:
detecting an event-related potential (ERP) in the user's response to presentation of the visual or auditory stimulus by searching within the set of EEG signals for at least one of:
an ERP-P3b waveform having a positive deflection in electrical potential within an ERP-P3b time-based search window following the presenting of the visual or auditory stimulus, or an ERP-N400 waveform having a negative deflection in electrical potential within an ERP-N400 time-based search window following the presenting of the visual or auditory stimulus; and
for each waveform present within the set of EEG signals, measuring a peak-amplitude value at the peak of that waveform and a peak-time value at the peak of that waveform relative to a time of presentation of the visual or auditory stimulus.
16. The HMD of claim 15 , wherein the operations further comprise:
communicating the level of drowsiness for the user with respect to the stimulus based on the measurements of the peak-amplitude and the peak-time value of the ERP-P3b waveform.
17. The HMD of claim 15 , wherein the operations further comprise:
communicating a level of attentiveness for the user with respect to the stimulus based on the measurements of the peak-amplitude and the peak-time value of the ERP-P3b waveform.
18. The HMD of claim 15 , wherein the operations further comprise:
communicating a level of comprehension for the user with respect to an information-bearing message of the stimulus based on the measurements of the peak-amplitude and the peak-time value of the ERP-N400 waveform.
19. The HMD of claim 11 , wherein the operations further comprise:
obtaining one or more of a Theta band, Alpha band, and Beta band, in which each band corresponds to a respective frequency range of the set of EEG signals; and
wherein the level of drowsiness for the user is further based on a combination of a measurement of amplitude and a regularity of frequency of the set of EEG signals within the one or more of the bands.
20. A non-transitory machine-readable medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
presenting a visual or auditory stimulus to a user of a head mounted device (HMD) worn on a head of the user, the HMD being configured to display augmented reality content in a transparent display of the HMD;
obtaining a set of electroencephalography (EEG) signals via a set of EEG electrodes of the HMD, each EEG electrode being configured to detect fluctuations of an electrical potential at a corresponding location relative to the head of the user in response to the visual or auditory stimulus being presented;
computing a level of drowsiness of the user based on the set of EEG signals; and
determining the augmented reality content in response to a change in the level of drowsiness.