System and method for enhanced training using a virtual reality environment and bio-signal data

1. A training apparatus comprising:

an input device and a wearable computing device with a bio-signal sensor and a display to provide an interactive continuous virtual reality (“VR”) environment for a user, the VR environment containing virtual elements, the bio-signal sensor receives bio-signal data from the user, the bio-signal sensor comprising a brainwave sensor;

the computing device having or in communication with a processor configured to:

as part of the interactive continuous VR environment, present content on the display where the content includes the virtual elements and has a VR event occurring within the interactive continuous VR environment, the VR event having one or more changes on the virtual elements in the VR environment, the VR event having desired user states, and desired effects;

continuously receive user manual inputs from the input device for user interaction with the virtual elements in the interactive continuous VR environment including during the VR event;

continuously receive the bio-signal data of the user from the bio-signal sensor during the VR event;

process the bio-signal data to determine user states of the user, including brain states, during the VR event, the user states are processed using a user profile stored in a data storage device accessible by the processor and the user states include brain states;

determine a user state score by comparing the user states of the user to the desired user states during the course of the VR event;

determine a performance score by comparing the effects to the desired effects during the course of the VR event; and

provide feedback to the user wherein the feedback is based on a combination of the user state score and the performance score,

wherein the user manual inputs are continuously received independent of the user state score.

2. The apparatus of claim 1 where the wearable computing device further comprises an inertial sensor and the bio-signal sensor further comprises a facial bio-signal sensor, wherein the facial sensor includes an EOG sensor, and where the bio-signal data further comprises data from the facial bio-signal sensor and the inertial sensor; the computing device is further configured to:

receive the bio-signal data from the EOG sensor and the inertial sensor for a user head and eye gaze direction

wherein the user states further comprises the user head and eye gaze direction, and the desired user states further comprises a desired user head and eye gaze direction.

3. The apparatus of claim 1 wherein the brain states comprises one or more of ability of operator to learn; prediction error; and emotional state leading to impaired thinking.

4. The apparatus of claim 1 further comprising presenting the user state score and the performance score synchronized with the content and the VR event to assist the user to better attain the desired user states and desired manual inputs on the input device.

5. The apparatus of claim 4 where

the computing device is further configured to:

revise the content in response to the feedback provided to the user where the user is further trained on the revised content.

6. The apparatus of claim 4 wherein the user state score further comprises failure brain states.

7. The apparatus of claim 1 where the display is a stereoscopic display.

8. The apparatus of claim 1 further comprising

a second display for presenting the content and a visual representation of the user states and manual inputs of the user in real time.

9. The apparatus of claim 1 wherein the computing device having or in communication with the processor is further configured to:

provide real time or near real time feedback to the user during the presentation of the content.

10. The apparatus of claim 1 , wherein the VR event is associated with event time data and a portion of bio-signal data is associated with bio-signal time data corresponding to the event time data, wherein the processor is configured to identify a portion of the bio-signal data based on the event time data and process the portion of the bio-signal data to determine the user states during the VR event, the bio-signal time data synchronized to the event time data.

11. The apparatus of claim 1 , wherein the VR event is associated with event time data and the bio-signal data is associated with bio-signal time data, and wherein the processor is configured to identify a time interval based on an expected response time for the VR event and the event time data, identify a portion of the bio-signal data based on the time interval and the bio-signal time data, and process the portion of the bio-signal data to determine the user states during the VR event, the bio-signal time data synchronized to the event time data.

12. A training method implemented using an input device and a wearable computing device having or in communication with a processor, a bio-signal sensor and a display to provide an interactive continuous virtual reality (“VR”) environment for a user, the VR environment containing a plurality of virtual elements, the bio-signal sensor receives bio-signal data from the user, the bio-signal sensor comprising a brainwave sensor; the training method comprising:

as part of the interactive continuous VR environment, presenting content on the display where the content has a VR event occurring within the interactive continuous VR environment, the VR event having one or more changes on at least a portion of the plurality of virtual elements in the VR environment, the VR event having desired user states, and desired effects;

continuously receiving user manual inputs from the input device for user interaction with the virtual elements in the interactive continuous VR environment including during the VR event;

continuously receiving the bio-signal data of the user from the bio-signal sensor during the VR event;

processing the bio-signal data to determine user states of the user, including brain states, during the VR event, the user states are processed using a user profile stored in a data storage device accessible by the processor and the user states include brain states;

determining a user state score by comparing the user states of the user to the desired user states during the course of the VR event;

determining a performance score by comparing the effects to the desired effects during the course of the VR event; and

providing feedback to the user wherein the feedback is based on a combination of the user state score and the performance score,

wherein the user manual inputs are continuously received independent of the user state score.

13. The training method of claim 12 , where the wearable computing device further comprises an inertial sensor and the bio-signal sensor further comprises a facial bio-signal sensor, wherein the facial sensor includes an EOG sensor, and where the bio-signal data further comprises data from the facial bio-signal sensor and the inertial sensor; the method further comprising

receiving the bio-signal data from the EOG sensor and the inertial sensor for a user head and eye gaze direction;

wherein the user states further comprises the user head and eye gaze direction, and the desired user states further comprises a desired user head and eye gaze direction.

14. The training method of claim 12 , wherein the brain states comprises one or more of ability of operator to learn; prediction error; and emotional state leading to impaired thinking.

15. The training method of claim 12 , further comprising post presenting the user state score and the performance score synchronized with the content and the VR event to assist the user to better attain the desired user states and desired manual inputs on the input device.

16. The training method of claim 15 , further comprising

revising the content in response to the feedback provided to the user where the user is further trained on the revised content.

17. The training method of claim 15 , wherein the user state score further comprises failure brain states.

18. The training method of claim 12 , where the display is a stereoscopic display.

19. The training method of claim 15 , further comprising

presenting the content and the user states and manual inputs of the user in real time on a second display.

20. The training method of claim 15 , further comprising

providing real time feedback to the user during the presentation of the content.

21. The apparatus of claim 1 , wherein the desired user state includes a distribution of desired brain states and wherein the determining of the user state score includes determining the deviation of the brain state from the distribution of desired brain states.

22. The apparatus of claim 1 , wherein the processor is further configured to dynamically update the user profile using the bio-signal data.