Generating communicative behaviors for anthropomorphic virtual agents based on user's affect
Systems and methods for automatically generating at least one of facial expressions, body gestures, vocal expressions, or verbal expressions for a virtual agent based on emotion, mood and/or personality of a user and/or the virtual agent are provided. Systems and method for determining a user's emotion, mood and/or personality are also provided.
1. A method for automatically generating at least one of facial expressions, body gestures, vocal expressions, or verbal expressions for a virtual agent, the method comprising:
receiving, by a computer, an emotion vector for a user, a mood vector for the user, and a personality vector of the user, wherein the emotion vector for the user, the mood vector for the user and the personality vector of the user are all for a particular utterance of the user;
determining, by a computer, an emotion vector for the virtual agent based on the emotion vector of the user;
determining, by the computer, a mood vector for the virtual agent based on i) the emotion vector for the virtual agent, and ii) the mood vector of the user;
determining, by the computer, a personality vector for the virtual agent based on i) the mood vector for the virtual agent, and ii) a personality vector of the user;
determining, by the computer, at least one of a facial expression, body gesture, vocal expression, or verbal expression for the virtual agent based on a content of the particular user utterance and at least one of the emotion vector for the virtual agent, the mood vector for the virtual agent and the personality vector for the virtual agent; and
applying, by the computer, the facial expression, body gesture, vocal expression, verbal expression, or any combination thereof to the virtual agent to produce control of the virtual agent's vocal expression, facial expression or both.
2. The method of claim 1 wherein the emotion vector, mood vector, and the personality vector are expressed as a pleasure, arousal, and dominance model.
3. The method of claim 1 wherein applying the facial expression, body gesture, vocal expression, verbal expression or any combination thereof to the virtual agent further comprise outputting by the virtual agent a virtual agent utterance.
4. The method of claim 1 wherein applying the facial expression body gesture, vocal expression, verbal expressions, or any combination thereof further comprises generating behavior markup language.
5. The method of claim 1 further comprising determining, by the computer, one or more utterances for the virtual agent based on the content of the particular user utterance and at least one of the emotion vector for the virtual agent, the mood vector for the virtual agent and the personality vector for the virtual agent.
6. The method of claim 5 wherein determining one or more utterances is further based on a profile of the user.
7. The method of claim 5 wherein determining one or more utterances further comprises for each of the one or more utterances, determining each word of the current utterance based on previously recorded interactions.
8. The method of claim 5 wherein determining one or more utterances further comprises for each of the one or more utterances, selecting a single utterance from a group of utterance candidates based on a prediction of the next utterance.
9. The method of claim 1 wherein determining an emotion vector for the virtual agent is further based on an emotion vector of the virtual agent for a current utterance of the virtual agent, and an emotion vector for the virtual agent's last utterance.
10. The method of claim 9 wherein determining the emotion vector for the virtual agent further comprises assigning, by the computer, a weight to at least one of i) the emotion vector of the user, ii) the emotion vector of the virtual agent for a current utterance of the virtual agent, and iii) the emotion vector for the virtual agent's last utterance, based on a type of the virtual agent, an administrative user input, or any combination thereof.
11. The method of claim 1 wherein determining a mood vector for the virtual agent is further based a mood vector for the virtual agent's last utterance.
12. The method of claim 11 wherein determining a mood vector for the virtual agent further comprises assigning a weight to at least one of i) the emotion vector for the virtual agent, ii) the mood vector of the user, and iii) the mood vector for the virtual agent's last utterance, based on a type of the virtual agent, an administrative user input, or combination thereof.
13. The method of claim 1 wherein determining a personality vector for the virtual agent is further based on a personality vector for the virtual agent's last utterance.
14. The method of claim 13 wherein determining a personality vector for the virtual agent further comprises assigning, by the computer, a weight to at least one of i) mood vector for the virtual agent, ii) the personality vector of the user, and iii) the personality vector for the virtual agent's last utterance based on a type of the virtual agent, an administrative user input, or combination thereof.
15. The method of claim 1 wherein determining a facial expression, body gesture, vocal expression, or verbal expression for the virtual agent is further based on content of an utterance of the virtual agent.
16. An article comprising a non-transitory computer readable storage medium containing program instructions for causing a computer to perform a method for automatically generating at least one of facial expressions, body gestures, vocal expressions, or verbal expressions for a virtual agent, the method comprising:
receive an emotion vector for a user, a mood vector for the user, and a personality vector of the user, wherein the emotion vector for the user, the mood vector for the user and the personality vector of the user are all for a particular utterance of the user;
determine an emotion vector for the virtual agent based on the emotion vector of the user;
determine a mood vector for the virtual agent based on i) the emotion vector for the virtual agent, and ii) the mood vector of the user;
determine a personality vector for the virtual agent based on i) the mood vector for the virtual agent, and ii) a personality vector of the user;
determine at least one of a facial expression, body gesture, vocal expression, or verbal expression for the virtual agent based on a content of the particular user utterance and at least one of the emotion vector for the virtual agent, the mood vector for the virtual agent and the personality vector for the virtual agent; and
apply the facial expression, body gesture, vocal expression, verbal expression, or any combination thereof to the virtual agent to produce control of the virtual agent's vocal expression, facial expression or both.
17. An apparatus comprising:
a processor configured to:
receive an emotion vector for a user, a mood vector for the user, and a personality vector of the user, wherein the emotion vector for the user, the mood vector for the user and the personality vector of the user are all for a particular utterance of the user;
determine an emotion vector for the virtual agent based on the emotion vector of the user;
determine a mood vector for the virtual agent based on i) the emotion vector for the virtual agent, and ii) the mood vector of the user;
determine a personality vector for the virtual agent based on i) the mood vector for the virtual agent, and ii) a personality vector of the user;
determine at least one of a facial expression, body gesture, vocal expression, or verbal expression for the virtual agent based on a content of the particular user utterance and at least one of the emotion vector for the virtual agent, the mood vector for the virtual agent and the personality vector for the virtual agent; and
apply the facial expression, body gesture, vocal expression, verbal expression, or any combination thereof to the virtual agent to produce control of the virtual agent's vocal expression, facial expression or both.