Asynchronous brain computer interface in AR using steady-state motion visual evoked potential

1. A method comprising:

receiving one or more requested stimuli data from a user application on a smart device;

receiving at least one of sensor data and other context data, wherein the sensor data includes environmental stimuli from a surrounding environment and the other context data includes data that is un-sensed;

transforming at least a portion of the requested stimuli data, into modified stimuli, based at least in part on at least one of the sensor data and the other context data;

presenting the modified stimuli and the environmental stimuli to the user with a rendering device configured to mix the modified stimuli and the environmental stimuli, thereby resulting in rendered stimuli;

receiving biosignals from the user, generated in response to the rendered stimuli, on a wearable biosignal sensing device;

classifying the received biosignals using a classifier based on the modified stimuli, resulting in a classified selection; and

returning the classified selection to the user application.

2. The method of claim 1 , further comprising, after receiving the biosignals from the user:

determining whether to send the received biosignals to the classifier by using at least one of:

the existence of an intentional control signal, wherein determination of the existence of the intentional control signal includes at least one of:

detecting a manual intention override signal from the smart device; and

determining, at least in part, from the received biosignals that the user is intending to fixate on at least one of the rendered stimuli; and

the absence of the intentional control signal;

on condition the intentional control signal exists:

sending the received biosignals, to the classifier; and

on condition the intentional control signal is absent:

continue receiving the received biosignals from the user.

3. The method of claim 1 , wherein the modified stimuli is based in part on determining a device context state using at least one of the sensor data and the other context data.

4. The method of claim 1 , wherein presenting the modified stimuli and the environmental stimuli to the user includes rendering the modified stimuli and the environmental stimuli using at least one of a visual device, a haptic device, and an auditory device sensed by the user.

5. The method of claim 1 , wherein the modified stimuli include steady-state motion visually evoked potential stimuli, and presenting the modified stimuli and the environmental stimuli to the user includes rendering the modified stimuli and the environmental stimuli on an augmented reality optical see-through (AR-OST) device associated with the smart device.

6. The method of claim 1 , wherein the at least one of the sensor data and the other context data includes at least one of:

environmental data, body-mounted sensor data, connected ambulatory device data, location specific connected device data, and network connected device data.

7. The method of claim 1 , further comprising:

receiving, by a cloud server, the classified selection from the classifier, the cloud server including:

a context manager;

a machine learning model, used by the smart device to facilitate classification of the received biosignals by the classifier; and

at least one model modification process for modifying the machine learning model;

receiving, by the context manager, at least one of current context state data and requests for other state data;

receiving, by the at least one model modification process, at least one of new state data and updated state data from the context manager; and

updating the machine learning model using the at least one model modification process and at least one of the classified selection, the new state data, and the updated state data.

8. The method of claim 7 , further comprising:

sending an updated machine learning model, from the cloud server to the smart device; and

transmitting the updated machine learning model to the classifier using a machine learning model transmission controller on the smart device.

9. The method of claim 8 , further comprising:

requesting a new machine learning model from the cloud server, by a context module on the smart device using the machine learning model transmission controller;

receiving, by the smart device, the new machine learning model from the cloud server; and

transmitting the new machine learning model to the classifier.

10. The method of claim 1 , further comprising a context manager on a cloud server, wherein the context manager provides additional context information to the smart device.

11. A system comprising:

a smart device;

a rendering device;

a wearable biosignal sensing device on a user;

a processor; and

a memory storing instructions that, when executed by the processor, configure the system to:

receive one or more requested stimuli data from a user application on the smart device;

receive at least one of sensor data and other context data, wherein the sensor data includes environmental stimuli from a surrounding environment and the other context data includes data that is un-sensed;

transform at least a portion of the requested stimuli data, into modified stimuli, based at least in part on at least one of the sensor data and the other context data;

present the modified stimuli and the environmental stimuli to the user with the rendering device configured to mix the modified stimuli and the environmental stimuli, thereby resulting in rendered stimuli;

receive biosignals from the user, generated in response to the rendered stimuli, on the wearable biosignal sensing device;

classify the received biosignals using a classifier based on the modified stimuli, resulting in a classified selection; and

return the classified selection to the user application.

12. The system of claim 11 , wherein the instructions further configure the system to, after receiving the biosignals from the user:

determine whether to send the received biosignals to the classifier by using at least one of:

the existence of an intentional control signal, wherein determination of the existence of the intentional control signal includes at least one of:

detect a manual intention override signal from the smart device; and

determine, at least in part, from received biosignals that the user is intending to fixate on at least one of the rendered stimuli; and

the absence of the intentional control signal;

on condition the intentional control signal exists:

send the received biosignals, to the classifier; and

on condition the intentional control signal is absent:

continue to receive the received biosignals from the user.

13. The system of claim 11 , wherein the modified stimuli is based in part on determining a device context state using at least one of the sensor data and the other context data.

14. The system of claim 11 , wherein presenting the modified stimuli and the environmental stimuli to the user includes rendering the modified stimuli and the environmental stimuli using at least one of a visual device, a haptic device, and an auditory device sensed by the user.

15. The system of claim 11 , wherein the modified stimuli include steady-state motion visually evoked potential stimuli, and presenting the modified stimuli and the environmental stimuli to the user includes rendering the modified stimuli and the environmental stimuli on an augmented reality optical see-through (AR-OST) device associated with the smart device.

16. The system of claim 11 , wherein the at least one of the sensor data and the other context data includes at least one of:

environmental data, body-mounted sensor data, connected ambulatory device data, location specific connected device data, and network connected device data.

17. The system of claim 11 , wherein the instructions further configure the system to:

receive, by a cloud server, the classified selection from the classifier,

the cloud server including:

a context manager;

a machine learning model, used by the smart device to facilitate classification of the received biosignals by the classifier; and

at least one model modification process for modifying the machine learning model;

receive, by the context manager, at least one of current context state data, and requests for other state data;

receive, by the at least one model modification process, at least one of new state data, and updated state data from the context manager; and

update the machine learning model using the at least one model modification process and at least one of the classified selection, the new state data, and the updated state data.

18. The system of claim 17 , wherein the instructions further configure the system to:

send an updated machine learning model, by the cloud server to the smart device; and

transmit the updated machine learning model to the classifier using a machine learning model transmission controller on the smart device.

19. The system of claim 18 , wherein the instructions further configure the system to:

request a new machine learning model from the cloud server, by a context module on the smart device using the machine learning model transmission controller;

receive, by the smart device, the new machine learning model from the cloud server; and

transmit the new machine learning model to the classifier.

20. A method comprising:

receiving one or more requested stimuli data from a user application on a smart device;

receiving at least one of sensor data and other context data, wherein the sensor data includes environmental stimuli from a surrounding environment and the other context data includes data that is un-sensed;

transforming at least a portion of the requested stimuli data, into modified stimuli, based at least in part on at least one of the sensor data and the other context data, wherein the modified stimuli include steady-state motion visually evoked potential stimuli;

presenting the modified stimuli and the environmental stimuli to the user with a rendering device configured to mix the modified stimuli and the environmental stimuli, thereby resulting in rendered stimuli, wherein presenting the modified stimuli and the environmental stimuli to the user includes at least one of:

rendering the modified stimuli and the environmental stimuli using at least one of a visual device, a haptic device, and an auditory device sensed by the user; and

rendering the modified stimuli and the environmental stimuli on an augmented reality optical see-through (AR-OST) device associated with the smart device;

receiving biosignals from the user, generated in response to the rendered stimuli, on a wearable biosignal sensing device;

determining whether to send the biosignals to a classifier by using at least one of:

the existence of an intentional control signal, wherein determination of the existence of the intentional control signal includes at least one of:

detecting a manual intention override signal from the smart device; and

determining, at least in part, from received biosignals that the user is intending to fixate on at least one of the rendered stimuli; and

the absence of the intentional control signal;

on condition the intentional control signal exists:

sending the received biosignals, to the classifier; and

on condition the intentional control signal is absent:

continue receiving the received biosignals from the user;

classifying the received biosignals using the classifier based on the modified stimuli, resulting in a classified selection; and

returning the classified selection to the user application.