Image-based instant messaging system for providing expressions of emotions
View Patent ↗Emotions can be expressed in the user interface for an instant messaging system based on the content of a received text message. The received text message is analyzed using a text-to-speech engine to generate phoneme data and wave data based on the text content. Emotion tags embedded in the message by the sender are also detected. Each emotion tag indicates the sender's intent to change the emotion being conveyed in the message. A mapping table is used to map phoneme data to viseme data. The number of face/lip frames required to represent viseme data is determined based on at least the length of the associated wave data. The required number of face/lip frames is retrieved from a stored set of such frames and used in generating an animation. The retrieved face/lip frames and associated wave data are presented in the user interface as synchronized audio/video data.
1. A method of generating an animation using stored face/lip frames corresponding to a plurality of emotions to express emotion in connection with text message, the method comprising:
analyzing the text message to generate phoneme data and wave data based on content of the text message;
mapping generated phoneme data to viseme data representing a particular emotion based on information identifying the particular emotion, wherein the information is associated with the text message;
calculating a needed number of the stored face/lip frames associated with the viseme data based on a length of the generated wave data; and
retrieving the calculated number of the stored face/lip frames to generate an animation associated with the generated wave data;
wherein the needed number of the face/lip frames is calculated as N×T×B, wherein N is a number of face/lip frames associated with the viseme data, T is a length of wave data of a current phoneme in seconds, and B is a bigram value corresponding to a frequency of a combination of the current phoneme and a preceding phoneme.
2. The method of claim 1 , further comprising presenting the generated wave data and the retrieved face/lip frames to a user as synchronized audio/video data.
3. The method of claim 1 , wherein the information identifying the particular emotion comprises at least one emotion tag embedded in the text message by a sender, and wherein analyzing the text message further comprises detecting the at least one emotion tag, each detected emotion tag representing the sender's intent to change the emotion being expressed in the message.
4. The method of claim 3 , wherein face/lip frames to be retrieved are identified in the at least one emotion tag embedded in the text message.
5. The method of claim 1 , wherein N=16.
6. A non-transitory computer readable medium having stored thereon computer usable program code for generating an animation using stored face/lip frames corresponding to a plurality of emotions to express emotion in connection with a text message, said computer usable program code comprising:
computer usable program code configured to analyze the text message to generate phoneme data and wave data based on content of the text message;
computer usable program code configured to map generated phoneme data to viseme data representing a particular emotion based on information identifying the particular emotion, wherein the information is associated with the text message;
computer usable program code configured to calculate a needed number of the stored face/lip frames associated with the viseme data based on a length of the generated wave data; and
computer usable program code configured to retrieve the calculated number of the stored face/lip frames to generate an animation associated with the generated wave data;
wherein said computer usable program code configured to calculate the needed number of the face/lip frames is configured to calculate the needed number of the face/lip frames as N×T×B, wherein N is a number of face/lip frames associated with the viseme data, T is a length of wave data of a current phoneme in seconds, and B is a bigram value corresponding to a frequency of a combination of the current phoneme and a preceding phoneme.
7. The non-transitory computer readable medium of claim 6 , further storing computer usable program code configured to present the generated wave data and the retrieved face/lip frames to a user as synchronized audio/video data.
8. The non-transitory computer readable medium of claim 6 ,
wherein the information identifying the particular emotion comprises at least one emotion tag embedded in the text message by a sender, and wherein the computer usable program code configured to analyze the text message further comprises computer usable program code configured to detect the at least one emotion tag, each detected emotion tag representing the sender's intent to change the emotion being expressed in the message.
9. The non-transitory computer readable medium of claim 8 , further storing computer usable program code configured to associate particular face/lip frames to be retrieved with the at least one detected emotion tag embedded in the text message.
10. The non-transitory computer readable medium of claim 6 , wherein N=16.
11. A visual speech system for generating an animation using stored face/lip frames corresponding to a plurality of emotions to express emotion in connection with a text message, the system comprising: a processor; and
at least one memory coupled to the processor, the at least one memory having stored thereon processor-executable instructions for:
analyzing the text message to generate phoneme data and wave data based on content of the text message;
mapping generated phoneme data to viseme data representing a particular emotion based on information identifying the particular emotion, wherein the information is associated with the text message;
calculating a needed number of the stored face/lip frames associated with the viseme data based on a length of the generated wave data; and
retrieving the calculated number of the stored face/lip frames to generate an animation associated with the generated wave data;
wherein the processor-executable instructions for calculating the needed number of the face/lip frames calculate the needed number of the face/lip frames as N×T×B, wherein N is a number of face/lip frames associated with the viseme data, T is a length of wave data of a current phoneme in seconds, and B is a bigram value corresponding to a frequency of a combination of the current phoneme and a preceding phoneme.
12. The visual speech system of claim 11 , further comprising a user interface for presenting the generated wave data and the retrieved face/lip frames to a user as synchronized audio/video data.
13. The visual speech system of claim 11 , wherein the information identifying the particular emotion comprises at least one emotion tag, and the at least one memory further has stored thereon processor-executable instructions for:
detecting the at least one emotion tag embedded in the text message by a sender, each detected emotion tag representing the sender's intent to change the emotion being expressed in the message.
14. The visual speech system of claim 13 , wherein face/lip frames to be retrieved are identified in the at least one emotion tag embedded in the text message.