Generating communicative behaviors for anthropomorphic virtual agents based on user's affect
Systems and methods for automatically generating at least one of facial expressions, body gestures, vocal expressions, or verbal expressions for a virtual agent based on emotion, mood and/or personality of a user and/or the virtual agent are provided. Systems and method for determining a user's emotion, mood and/or personality are also provided.
1. A method to control output of a virtual agent based on a user's affect, the computerized-method comprising:
receiving, by a computer, an utterance of a user;
determining, by the computer, an emotion vector of the user based a content of the utterance, wherein determining the emotion vector of the user comprises:
i) determining a sentiment score of the utterance by determining a sentiment score for each word in the utterance, the sentiment score based on one or more predefined rules,
ii) determining a dialogue performance score based on one or more received dialogue performance metrics,
iii) determining a vocal emotion score based on a received vocal characteristic of the user,
iv) determining a facial emotion score based on a received facial expression of the user, and
v) determining the emotion vector by averaging the sentiment score, the vocal score, the facial emotion score, and the dialogue performance score;
determining, by the computer, a mood vector of the user based on the emotion vector of the user;
determining, by the computer, a personality vector of the user based on the mood vector of the user;
determining, by the computer, at least one of a facial expression, body gesture, vocal expression, or verbal expression for the virtual agent based on a content of the utterance of the user and at least one of the emotion vector for user, the mood vector for the user and the personality vector for the user; and
applying, by the computer, the facial expression, body gesture, vocal expression, verbal expression, or any combination thereof to the virtual agent to produce control of the virtual agent's vocal expression, facial expression or both.
2. The method of claim 1 wherein the mood vector of the user is further based on a previous mood vector of the user.
3. The method of claim 1 wherein the personality vector of the user is further based on a previous personality vector of the user.
4. The method of claim 1 further comprising receiving a vocal characteristic or a facial expression of the user.