Networked virtual reality training systems and methods
Disclosed herein are embodiments for managing a virtual reality (VR) training exercise via a management server. The management server outputs a graphical dashboard including one or more skill nodes, and selects one or more software agents associated with the skill nodes. The management server provides the software agents to at least one host computing system communicatively coupled to a near-to-eye display device. The near-to-eye display device is configured to display a virtual three dimensional (3D) training environment including a plurality of interactive 3D virtual objects. The software agents are configured to collect VR observables data while the trainee performs actions within the virtual 3D training environment. Based on the VR observables data collected, the management server determines that one or more skills have been demonstrated during the training exercise, and updates the one or more skill nodes to graphically indicate the one or more skills demonstrated by the trainee.
1 . A method comprising:
outputting, by a server and for display, a graphical dashboard associated with a training exercise, wherein the graphical dashboard further includes a directed graph comprising:
a plurality of metrics to be collected during execution of the training exercise;
a plurality of skill nodes that represent a plurality of skills to be demonstrated by a trainee during the training exercise;
a plurality of learning objectives;
indications of a first set of hierarchical relationships between the plurality of metrics and the plurality of skill nodes and a second set of hierarchical relationships between the plurality of skill nodes and the plurality of learning objectives; and
a mapping of a plurality of software agents to the plurality of metrics;
selecting, by the server, the plurality of software agents that are associated with the the plurality of skill nodes, the plurality of software agents configured to collect virtual reality (VR) observables data for calculating the plurality of metrics corresponding to the plurality of skill nodes;
providing, by the server and to at least one host computing system, an indication of the plurality of software agents that are executed during the training exercise;
providing, by the server and to the at least one host computing system, one or more metric parameters to configure the plurality of software agents to calculate the plurality of metrics for determining whether the plurality of skills have been demonstrated, wherein each of the at least one host computing system is communicatively coupled to a near-to-eye display device configured to display a virtual three dimensional (3D) training environment including a plurality of interactive 3D virtual objects, wherein the plurality of software agents are configured to collect the VR observables data from the at least one host computing system while the trainee performs one or more trainee input actions within the virtual 3D training environment displayed by the near-to-eye display device, wherein the VR observables data match the one or more trainee input actions performed within the virtual 3D training environment to a state of each of one or more interactive 3D virtual objects of the plurality of interactive 3D virtual objects;
receiving, by the server, the VR observables data collected by the plurality of software agents during execution and the plurality of metrics calculated from the VR observables data;
automatically monitoring, by the server, the trainee input actions performed within the virtual 3D training environment using the VR observables data received from the plurality of software agents and the plurality of metrics calculated by the plurality of software agents from the VR observables data to determine whether the plurality of skills represented by the plurality of skill nodes have been demonstrated by the trainee during the training exercise;
responsive to determining that the one or more of the plurality of skills have been demonstrated, updating, by the server and for display, the graphical dashboard to indicate that the one or more of the plurality of skills have been demonstrated by the trainee during the training exercise; and
responsive to determining that the one or more of the plurality of skills indicate that the trainee met a learning objective of the plurality of learning objectives, updating, by the server and for display, the graphical dashboard to indicate that the learning objective of the plurality of learning objectives has been met.
2 . The method of claim 1 , wherein the VR observables data match the one or more trainee input actions performed within the virtual 3D training environment to magnification of at least one interactive 3D virtual object of the plurality of interactive 3D virtual objects.
3 . The method of claim 1 , wherein the VR observables data match the one or more trainee input actions performed within the virtual 3D training environment to level of detail of at least one interactive 3D virtual object of the plurality of interactive 3D virtual objects.
4 . The method of claim 1 , wherein the VR observables data match the one or more trainee input actions performed within the virtual 3D training environment to orientation of at least one interactive 3D virtual object of the plurality of interactive 3D virtual objects.
5 . The method of claim 1 , wherein the VR observables data track the one or more trainee input actions performed within the virtual 3D training environment in six degrees of freedom to determine one or more positions and orientations within the virtual 3D training environment associated with each trainee input action.
6 . The method of claim 1 , wherein the state of each of the one or more interactive 3D virtual objects of the plurality of interactive 3D virtual objects includes the plurality of metrics for the determining whether the plurality of skills have been demonstrated.
7 . The method of claim 6 , wherein the state of each respective interactive 3D virtual object of the one or more interactive 3D virtual objects comprises at least one of the state of the respective interactive 3D virtual object at a point in time or a change of state of the respective interactive 3D virtual object at a point in time or during a period of time.
8 . The method of claim 1 , wherein the VR observables data match the one or more trainee input actions performed within the virtual 3D training environment to a trajectory prediction model that outputs, for each trainee input action one or more of a heading, an angle, a course, and a derived velocity.
9 . The method of claim 1 , wherein the one or more skill nodes graphically indicate that the plurality of skills have not yet been demonstrated by the trainee, wherein the step of updating the graphical dashboard comprises updating the plurality of skill nodes to graphically indicate that the plurality of skills have been demonstrated by the trainee during the training exercise.
10 . The method of claim 1 , wherein the training exercise comprises a team exercise for a first team and a second team, wherein the trainee is a member of the first team, wherein the one or more skill nodes represent the one or more skills to be demonstrated by the first team during the training exercise.
11 . The method of claim 1 , wherein the near-to-eye display device is a head mounted VR display, and the virtual 3D training environment is an immersive virtual environment.
12 . The method of claim 1 , further comprising at least one tracked input device communicatively coupled with the at least one host computing system, wherein the at least one tracked input device is configured to transmit trainee inputs to the at least one host computing system to generate the one or more trainee input actions within the virtual 3D training environment.
13 . A system comprising:
a processor;
non-transitory machine-readable memory that stores one or more skill nodes, one or more representative training environments, and a plurality of interactive 3D virtual objects;
one or more host computing systems communicatively coupled to the processor; and
a near-to-eye display device communicatively coupled to at least one host computing system of the one or more host computing systems,
wherein the processor in communication with the non-transitory machine-readable memory and the at least one host computing system executes a set of instructions instructing the processor to:
output, for display, a graphical dashboard associated with a training exercise, wherein the graphical dashboard further includes a directed graph comprising:
a plurality of metrics to be collected during execution of the training exercise;
a plurality of skill nodes that represent a plurality of skills to be demonstrated by a trainee during the training exercise;
a plurality of learning objectives;
indications of a first set of hierarchical relationships between the plurality of metrics and the plurality of skill nodes and a second set of hierarchical relationships between the plurality of skill nodes and the plurality of learning objectives; and
a mapping of a plurality of software agents to the plurality of metrics;
select the plurality of software agents that are associated with the plurality of skill nodes, wherein the plurality of software agents are configured to collect virtual reality (VR) observables data for calculating the plurality of metrics corresponding to the plurality of skill nodes from the at least one host computing system while the trainee performs one or more trainee input actions within a virtual three-dimensional (3D) training environment displayed by the near-to-eye display device, wherein the VR observables data match the one or more trainee input actions performed within the virtual 3D training environment to a state of each of one or more interactive 3D virtual objects of the plurality of interactive 3D virtual objects;
transmit to the at least one host computing system for execution during the training exercise, an indication of the plurality of software agents that are executed during the training exercise;
transmit to the at least one host computing system one or more metric parameters to configure the plurality of software agents to calculate the plurality of metrics for determining whether the plurality of skills have been demonstrated;
receive from the at least one host computing system the VR observables data collected by the plurality of software agents during execution and the plurality of metrics calculated from the VR observables data;
automatically monitor the trainee input actions performed within the virtual 3D training environment using the VR observables data received from the plurality of software agents and the plurality of metrics calculated by the plurality of software agents from the VR observables data to determine whether the plurality of skills represented by the plurality of skill nodes have been demonstrated by the trainee during the training exercise; and
responsive to determining that the plurality of skills have been demonstrated, update the graphical dashboard to indicate that the plurality of skills have been demonstrated by the trainee during the training exercise; and
responsive to determining that the one or more of the plurality of skills indicate that the trainee met a learning objective of the plurality of learning objectives, update the graphical dashboard to indicate that the learning objective of the plurality of learning objectives has been met.
14 . The system of claim 13 , wherein the VR observables data match the one or more trainee input actions performed within the virtual 3D training environment to one or both magnification and level of detail of at least one interactive 3D virtual object of the plurality of interactive 3D virtual objects.
15 . The system of claim 13 , wherein the VR observables data match the one or more trainee input actions performed within the virtual 3D training environment to orientation of at least one interactive 3D virtual object of the plurality of interactive 3D virtual objects.
16 . The system of claim 13 , wherein the near-to-eye display device is a head mounted VR display, and the virtual 3D training environment is an immersive virtual environment.
17 . The system of claim 13 , further comprising at least one tracked input device communicatively coupled with the at least one host computing system, wherein the at least one tracked input device is configured to transmit trainee inputs to the at least one host computing system to generate the one or more trainee input actions within the virtual 3D training environment.
18 . The system of claim 13 , wherein at least one tracked input device comprises one or more of a hand-held keypad, a first device for tracking movements of the trainee's arms and hands, a cyberglove, a second device for tracking movements of the trainee's retina or pupil, a touch gesture sensor, a tactile input device with reaction force generator, and a camera.
19 . The method of claim 1 , wherein a further learning objective of the plurality of learning objectives is supported by a first skill and a second skill of the plurality of skills, the method further comprising:
responsive to determining that the first skill has been demonstrated and the second skill has not been demonstrated, updating, by the server and for display, the graphical dashboard to indicate that the further learning objective of the plurality of learning objectives has not been met.
20 . The method of claim 1 , wherein the plurality of software agents include one or more hypervisor agents to deploy onto virtual machine platforms hosting at least one virtual machine of the at least one host computing system.