Brain-computer interface with adaptations for high-speed, accurate, and intuitive user interactions
Embodiments described herein relate to systems, devices, and methods for use in the implementation of a brain-computer interface that integrates real-time eye-movement tracking with brain activity tracking to present and update a user interface that is strategically designed for high speed and accuracy of human—machine interaction. Embodiments described herein also relate to the implementation of a hardware agnostic brain-computer interface with specific user interface adaptations to enable high-speed, intuitive, and accurate user manipulation of applications and/or machines.
1 . An apparatus, comprising:
a display configured to present a control interface to a user;
an eye-tracking device configured to record eye-movement signals associated with the user;
a neural recording device configured to record neural signals associated with the user;
an interfacing device operatively coupled to the display, the eye-tracking device, and the neural recording device, the interfacing device including:
a memory; and
a processor operatively coupled to the memory and configured to:
receive the eye-movement signals from the eye-tracking device and the neural signals from the neural recording device;
generate and present a stimulus, via the control interface and to the user, the stimulus including a set of control items associated with a set of actions;
determine weights for the eye movement signals and the neural signals based on a state of the control interface, including the stimulus;
determine a point of focus of the user based on the eye-movement signals and the neural signals using the weights, the point of focus being associated with at least one control item from the set of control items;
determine, based on the point of focus and the at least one control item, an action intended by the user; and
implement the action intended by the user.
2 . The apparatus of claim 1 , wherein:
the eye-tracking device includes an optical sensor,
the neural signals include electroencephalography (EEG) signals including at least one of a visually evoked potential, a sensory evoked potential, a motor imagery signal, an Event Related Potential (ERP), a sensorimotor rhythm, an event related desynchronization (ERD), an event related synchronization (ERS), a slow cortical potential (SCP), and a brain state dependent signal, and
the processor is further configured to integrate the eye-movement signals and the EEG signals to determine the point of focus of the user.
3 . The apparatus of claim 1 , wherein the display is configured to present the control interface as a three-dimensional space.
4 . The apparatus of claim 1 , wherein the stimulus includes at least one of a visual stimulus, an auditory stimulus, vestibular stimulus, and a haptic stimulus.
5 . The apparatus of claim 1 , wherein the set of control items are associated with controlling navigation through a virtual environment.
6 . The apparatus of claim 5 , wherein the set of control items include: a first subset of control items associated with controlling a velocity of virtual movement of the user, and a second subset of control items associated with controlling a direction of virtual movement of the user.
7 . The apparatus of claim 1 , wherein the display is further configured to present a view of a real-world environment and is configured to present the control interface over the view of the real-world environment.
8 . The apparatus of claim 1 , wherein:
the set of control items is a first set of control items, the stimulus is a first stimulus,
the display is further configured to present a view of a real-world environment and is configured to present the control interface over the view of the real-world environment, and
the processor is further configured to:
receive a set of images associated with the projected-real-world environment;
analyze the set of images to identify at least one machine operatively connected to the processor;
obtain information of a set of actions associated with the machine;
generate a second set of control items based on the information; and
present, via the control interface and to the user, a second stimulus including the second set of control items.
9 . A non-transitory processor-readable medium storing code representing instructions to be executed by a processor, the instructions comprising code to cause the processor to:
generate a control interface configured to be manipulated, by a user, to perform a set of actions;
generate and present a stimulus, via the control interface and to the user, the stimulus including a set of control items associated with a set of actions;
receive information from an eye-tracking device and information from a neural recording device, the eye-tracking device and the neural recording device configured to monitor behavior of the user;
determine weights for the information from the eve-tracking device and the information from the neural recording device based on a state of the control interface, including the stimulus;
determine, based on the information from the eye-tracking device and the information from the neural recording device and using the weights, a point of focus of the user;
identify at least one control item from the set of control items associated with the point of focus of the user; and
determine, based on the at least one control item, an action intended by the user.
10 . The non-transitory processor-readable medium of claim 9 , wherein the instructions further comprising code to cause the processor to execute the action intended by the user.
11 . The non-transitory processor-readable medium of claim 9 , wherein the instructions further comprising code to cause the processor to:
present to the user an indication of the action intended by the user for confirmation by the user; and
in response to receiving a signal representing a confirmation by the user, execute the action intended by the user.
12 . The non-transitory processor-readable medium of claim 9 , wherein the information received from the eye-tracking device includes eye-movement signals associated with the user, and
the information received from the neural recording device includes neural signals associated with the user, the neural signals including electroencephalography (EEG) signals, the EEG signals including at least one of a visually evoked potential, a sensory evoked potential, a motor imagery signal, an Event Related Potential (ERP), a sensorimotor rhythm, an event related desynchronization (ERD), an event related synchronization (ERS), a slow cortical potential (SCP), and a brain state dependent signal, and
the code to cause the processor to determine the point of focus includes code to cause the processor to integrate the eye-movement signals and the EEG signals to determine the point of focus of the user.
13 . The non-transitory processor-readable medium of claim 9 , wherein the code to cause the processor to present the stimulus includes code to cause the processor to present the set of control items in a three dimensional virtual space defined by the control interface.
14 . The non-transitory processor-readable medium of claim 9 , wherein the instructions further comprise code to cause the processor to group the set of control items into a set of groups, the code to cause the processor to present the stimulus includes code to cause the processor to present the set of the control items as grouped in the set of groups.
15 . The non-transitory processor-readable medium of claim 9 , wherein the set of control items includes a draggable control item, the draggable control item configured to be manipulated based on eye-movement signals associated with the user, the eye-movement signals included in information received from the eye-tracking device.
16 . The non-transitory processor-readable medium of claim 9 , wherein the at least one control item is a first control item and the point of focus of the user is at a first time, the instructions further comprising code to cause the processor to:
identify the first control item as a draggable control item, the draggable control item configured to be manipulated based on eye-movement signals associated with the user and included in the information received from the eye-tracking device;
determine a point of focus of the user at a second time after the first time;
move the draggable control item to a location associated with the point of focus of the user at the second time;
identify a second control item associated with the point of focus of the user at the second time, the second control item being different from the draggable control item;
determine, based on the second control item, the action intended by the user.
17 . The non-transitory processor-readable medium of claim 9 , wherein the at least one control item is a first control item and the point of focus of the user is at a first time,
the set of control items including a set of sticky control items, the sticky control items being configured to be manipulated based on eye-movement signals associated with the user, the eye-movement signals included in the information received from the eye-tracking device, the instructions further comprising code to cause the processor to:
provide a grabber object included in the control interface, the grabber object being configured to manipulate the sticky control items;
identify the first control item as a sticky control item;
associate, based on the sticky control item being associated with the point of focus of the user at the first time, the sticky control item with the grabber object;
determine a point of focus of the user at a second time after the first time,
identify a second control item associated with the point of focus of the user at the second time, the second control item being different from the sticky control item;
dissociate the sticky control item from the grabber object and associate the sticky control item with the second control item; and
determine, based on the association between the sticky control item and the second control item, the action intended by the user.
18 . A method comprising:
generating a control interface configured to be manipulated, by a user, to perform a set of actions;
generating and presenting a stimulus, via the control interface and to the user, the stimulus including a set of control items associated with a set of actions;
receiving information from an eye-tracking device and information from a neural recording device, the eye-tracking device and the neural recording device configured to monitor behavior of the user;
determining weights for the information from the eye-tracking device and the information from the neural recording device based on a state of the control interface, including the stimulus;
determining, based on the information from the eye-tracking device and the information from the neural recording device and using the weights, a point of focus of the user;
identifying at least one control item from the set of control items associated with the point of focus of the user; and
determining, based on the at least one control item, an action intended by the user.
19 . The method of claim 18 , wherein determining the point of focus of the user comprises:
extracting an Event Related Potential (ERP) from the information from the neural recording device; and
correlating occurrence of the ERP with the presentation of a particular stimulus in time and space to identify a causal relationship.
20 . The method of claim 18 , wherein determining the point of focus of the user comprises:
combining oculomotor data from the eye-tracking device with data from visually evoked neural activity from the neural recording device, and with strategic presentation of stimuli in the control interface for processing as an ensemble.
21 . The method of claim 20 , wherein weights for each source signal in the ensemble are used to combine the source signals.
22 . The method of claim 20 , wherein the ensemble is processed using a bagging ensemble classifier algorithm.