Communication and skills training using interactive virtual humans
A system for providing interaction between a virtual human and a user, the system comprising: a tangible interface providing a physical interface between the user and the virtual human, an imaging system directed towards the physical interface to provide images of the user interacting with the tangible interface; a tracking system tracking at least one position or the user; a microphone capturing speech from the user; a simulation system receiving inputs from the tangible interface, the imaging system, the tracking system and the microphone, the simulation system generating output signals corresponding to the virtual human; and a display presenting the output signals to the user.
1. A training system for providing a virtual human for training a user, the virtual human comprising an image of a human exhibiting one or more actions, the system comprising:
a tangible interface comprising a model of at least a portion of a human and at least one sensor configured to sense a physical interaction between the user and the model;
an imaging system directed towards the tangible interface and configured to capture images of the user interacting with the tangible interface;
a tracking system tracking at least one position of the user;
a microphone capturing speech from the user;
a simulation system receiving inputs from the tangible interface indicating the sensed physical interaction, the imaging system, the tracking system, and the microphone, wherein the simulation system is configured to process an input from the microphone as communication from the user to the virtual human and generate an output representing one or more actions of the virtual human responding to an interaction as indicated by the communication from the user to the virtual human in combination with the sensed physical interaction such that the output of the simulation system is based on both the sensed physical interaction with the model and the communication representing physical and vocal interaction with a human represented by the virtual human; and
a display coupled to the simulation system configured to receive the output and display to the user the virtual human exhibiting the one or more actions based on the output, whereby a mixed reality environment is provided.
2. The training system of claim 1 wherein:
the display is a head mounted display.
3. The training system of claim 1 wherein:
the at least one sensor of the tangible interface includes a plurality of force sensors providing force signals as the inputs to the simulation system.
4. The training system of claim 1 wherein:
the tracking system comprises a camera for tracking a plurality of locations of a plurality of markers worn by the user.
5. The training system of claim 4 wherein:
the camera is an infrared camera and the markers are infrared markers.
6. The training system of claim 4 wherein:
the tracking system is configured to track the user's head; and
the simulation system is configured to generate the output such that the virtual human is rendered on the display with a gaze directed at the user's head.
7. The training system of claim 1 further comprising:
a visualizer component configured to render, based on the images of the user interacting with the tangible interface, a representation of the user performing a task involving interaction with a human on the display.
8. The training system of claim 7 wherein:
the visualizer component generates on the display a visualization screen depicting the interaction between the user and the virtual human in a virtual environment.
9. The training system of claim 8 wherein:
the visualizer component generates on the display a transcript screen depicting a text transcript of the interaction between the user and the virtual human, the text transcript including a timeline.
10. The training system of claim 9 wherein:
the visualizer component generates on the display a graph of a factor plotted versus time, the factor comprising a metric of the interaction between the user and the virtual human.
11. The training system of claim 10 wherein:
the factor plotted versus time is an amount by which a body of the user leans.
12. The training system of claim 10 wherein:
the factor plotted versus time is a frequency of occurrence of a topic in a discussion between the user and the virtual human.
13. The training system of claim 1 , wherein the tangible interface further comprises clothes configured to be worn on the model.
14. The training system of claim 1 , wherein the virtual human on the display is positioned to be visible to the user when the user is positioned to physically interact with the model.
15. The training system of claim 7 , wherein the visualizer component is configured to incorporate the captured images of the user interacting with the tangible interface into the rendered representation on the display.
16. The training system of claim 15 , wherein a portion of the user is in the captured images and the rendered representation on the display.