Conversation-driven character animation
A system for producing conversation-driven character animation includes a computing platform having processing hardware and a system memory storing software code, the software code including multiple trained machine learning (ML) models. The processing hardware executes the software code to obtain a conversation understanding feature set describing a present state of a conversation between a digital character and a system user, and to generate an inference, using at least a first trained ML model of the multiple trained ML models and the conversation understanding feature set, the inference including labels describing a predicted next state of a scene within the conversation. The processing hardware further executes the software code to produce, using at least a second trained ML model of the multiple trained ML models and the labels, an animation stream of the digital character participating in the predicted next state of the scene within the conversation.
1. A system comprising:
a computing platform having a processing hardware, a plurality of sensors and a system memory storing a software code, the software code including a plurality of trained machine learning (ML) models;
the processing hardware configured to execute the software code to:
generate a conversation understanding feature set using at least a first trained ML model of the plurality of trained ML models and sensor data received from the plurality of sensors, the conversation understanding feature set describing a present state of a conversation between a digital character and a system user;
generate an inference, using at least a second trained ML model of the plurality of trained ML models and the conversation understanding feature set, the inference comprising a plurality of labels describing a predicted next state of a scene within the conversation; and
produce, using at least a third trained ML model of the plurality of trained ML models and the plurality of labels, an animation stream of the digital character participating in the predicted next state of the scene within the conversation.
2. The system of claim 1 , wherein the predicted next state of the scene includes a background, a lighting scheme, and one or more props.
3. The system of claim 1 , wherein the predicted next state of the scene includes at least one of a whole body movement, a gesture, or a facial expression of the digital character.
4. The system of claim 1 , wherein the system further comprises a display, and wherein the processing hardware is further configured to execute the software code to:
render the animation stream on the display.
5. The system of claim 4 , wherein the animation stream is rendered as a three-dimensional (3D) animation.
6. The system of claim 4 , wherein the animation stream is rendered as a two-dimensional (2D) animation.
7. The system of claim 1 , wherein the conversation understanding feature set includes voice analysis data and identifies at least one of an intent of the system user, a sentiment of the system user, or a sentiment of the digital character.
8. The system of claim 1 , wherein producing the animation stream comprises selecting a sequence of video frames from a database of pre-existing animation.
9. The system of claim 1 , wherein producing the animation stream comprises generating a sequence of video frames.
10. The system of claim 1 , wherein the animation stream further comprises at least one of synthesized speech by the digital character or one or more haptic effects.
11. A method for use by a system including a computing platform having a processing hardware, a plurality of sensors and a system memory storing a software code, the software code including a plurality of trained machine learning (ML) models:
generating, by the software code executed by the processing hardware, a conversation understanding feature set using at least a first trained ML model of the plurality of trained ML models and sensor data received from the plurality of sensors, the conversation understanding feature set describing a present state of a conversation between a digital character and a system user;
generating an inference, by the software code executed by the processing hardware and using at least a second trained ML model of the plurality of trained ML models and the conversation understanding feature set, the inference comprising a plurality of labels describing a predicted next state of a scene within the conversation; and
producing, by the software code executed by the processing hardware and using at least a third trained ML model of the plurality of trained ML models and the plurality of labels, an animation stream of the digital character participating in the predicted next state of the scene within the conversation.
12. The method of claim 11 , wherein the predicted next state of the scene includes a background, a lighting scheme, and one or more props.
13. The method of claim 11 , wherein the predicted next state of the scene includes at least one of a whole body movement, a gesture, or a facial expression of the digital character.
14. The method of claim 11 , wherein the digital character is depicted in the animation stream using a skeleton model and mesh overlay enabling use of the skeleton model with different mesh overlies corresponding respectively to different digital characters.
15. The method of claim 14 , wherein the skeleton model comprises a dense pose model.
16. The method of claim 11 , wherein the system further comprises a display, the method further comprising:
rendering, by the software code executed by the processing hardware, the animation stream on the display.
17. The method of claim 16 , wherein the animation stream is rendered as a three-dimensional (3D) animation.
18. The method of claim 11 , wherein the conversation understanding feature set includes voice analysis data and identifies at least one of an intent of the system user, a sentiment of the system user, or a sentiment of the digital character.
19. The method of claim 11 , wherein producing the animation stream comprises selecting a sequence of video frames from a database of pre-existing animation.
20. The method of claim 11 , wherein producing the animation stream comprises generating a sequence of video frames.
21. The method of claim 11 , wherein the animation stream further comprises at least one of synthesized speech by the digital character or one or more haptic effects.
22. A method for use by a system including a computing platform having a processing hardware and a system memory storing a software code, the software code including a plurality of trained machine learning (ML) models:
obtaining, by the software code executed by the processing hardware, a conversation understanding feature set describing a present state of a conversation between a digital character and a system user;
generating an inference, by the software code executed by the processing hardware and using at least a first trained ML model of the plurality of trained ML models and the conversation understanding feature set, the inference comprising a plurality of labels transformed into two-dimensional (2D) or three-dimensional (3D) skeletal keypoints for a skeleton model of the digital character, the plurality of labels describing a predicted next state of a scene within the conversation; and
producing, by the software code executed by the processing hardware and using at least a second trained ML model of the plurality of trained ML models and the plurality of labels, an animation stream of the digital character participating in the predicted next state of the scene within the conversation.
23. The method of claim 22 , wherein the system further comprises a display, the method further comprising:
rendering, by the software code executed by the processing hardware, the animation stream on the display.
24. The method of claim 22 , wherein the system further comprises a plurality of sensors, the method further comprising:
generating the conversation understanding feature set, by the software code executed by the processing hardware and using at least a third trained ML model of the plurality of trained ML models and sensor data received from the plurality of sensors.