Multi-level emotional enhancement of dialogue
A system for emotionally enhancing dialogue includes a computing platform having processing hardware and a system memory storing a software code including a predictive model. The processing hardware is configured to execute the software code to receive dialogue data identifying an utterance for use by a digital character in a conversation, analyze, using the dialogue data, an emotionality of the utterance at multiple structural levels of the utterance, and supplement the utterance with one or more emotional attributions, using the predictive model and the emotionality of the utterance at the multiple structural levels, to provide one or more candidate emotionally enhanced utterance(s). The processing hardware further executes the software code to perform an audio validation of the candidate emotionally enhanced utterance(s) to provide a validated emotionally enhanced utterance, and output an emotionally attributed dialogue data providing the validated emotionally enhanced utterance for use by the digital character in the conversation.
1. A system comprising:
a computing platform having a speech synthesizer, a processing hardware and a system memory storing a software code, the software code including at least one of a trained machine learning (ML) model configured to function as an autoregressive generator or a stochastic model trained using unsupervised or semi-supervised learning;
the processing hardware configured to execute the software code to:
receive dialogue data identifying an utterance for use by a digital character in a conversation;
analyze, using the dialogue data, an emotionality of the utterance at a plurality of structural levels of the utterance;
supplement the utterance with one or more emotional attributions, using the at least one of the trained ML model or the stochastic model and the emotionality of the utterance at the plurality of structural levels, to provide a plurality of candidate emotionally enhanced utterances;
perform an audio validation of at least some of the plurality of candidate emotionally enhanced utterances to provide a validated emotionally enhanced utterance including a non-verbal vocalization, wherein the audio validation identifies the validated emotionally enhanced utterance as having a best audio quality of the at least some of the plurality of candidate emotionally enhanced utterances;
output an emotionally attributed dialogue data providing the validated emotionally enhanced utterance for use by the digital character in the conversation; and
synthesize, using the speech synthesizer and the emotionally attributed dialogue data, the validated emotionally enhanced utterance to generate a synthesized speech for utterance by the digital character.
2. The system of claim 1 , wherein the digital character is configured to utter the synthesized speech.
3. The system of claim 1 , wherein the plurality of structural levels of the utterance comprise at least two of token level, a phrase level, or an entire utterance level.
4. The system of claim 1 , wherein the predictive model comprises the trained ML model, and wherein the trained ML model comprises transformer-based token insertion ML model.
5. The system of claim 4 , wherein the transformer-based token insertion ML model takes as input one or more of: a global sentiment of the dialogue data, a global mood of the dialogue data, a sentiment by phrase within the dialogue data, a sentiment by token in the dialogue data, a given character embedding, or extracted entities from the dialogue data.
6. The system of claim 1 , wherein the one or more emotional attributions identify one or more of a prosodic variation, or a word rate.
7. The system of claim 1 , wherein the conversation is between the digital character and a user of the system, and wherein the processing hardware is further configured to execute the software code to:
obtain a user profile of the user, the user profile including a user history of the user; and
analyze the emotionality of the utterance further using the user profile.
8. The system of claim 1 , wherein the digital character is associated with a character persona, and wherein the processing hardware is further configured to execute the software code to:
obtain the character persona; and
analyze the emotionality of the utterance further using the character persona.
9. The system of claim 1 , wherein the processing hardware is further configured to execute the software code to:
determine, using another predictive model of the software code, a quality score for each of the plurality of candidate emotionally enhanced utterances to provide a plurality of quality scores corresponding respectively to the plurality of candidate emotionally enhanced utterances; and
identify, based on the plurality of quality scores, the at least some of the plurality of candidate emotionally enhanced utterances for the audio validation.
10. The system of claim 1 , wherein the dialogue data and the emotionally attributed dialogue data comprise text.
11. The system of claim 1 , wherein the validated emotionally enhanced utterance further include a non-verbal filler.
12. A method for use by a system including a computing platform having a speech synthesizer, a processing hardware and a system memory storing a software code, the software code including at least one of a trained machine learning (ML) model configured to function as an autoregressive generator or a stochastic model trained using unsupervised or semi-supervised learning:
receiving, by the software code executed by the processing hardware, dialogue data identifying an utterance for use by a digital character in a conversation;
analyzing, by the software code executed by the processing hardware and using the dialogue data, an emotionality of the utterance at a plurality of structural levels of the utterance;
supplementing the utterance with one or more emotional attributions, by the software code executed by the processing hardware and using the at least one of the trained ML model or the stochastic model and the emotionality of the utterance at the plurality of structural levels, to provide a plurality of candidate emotionally enhanced utterances;
performing, by the software code executed by the processing hardware, an audio validation of at least some of the plurality of candidate emotionally enhanced utterances to provide a validated emotionally enhanced utterance including a non-verbal vocalization, wherein the audio validation identifies the validated emotionally enhanced utterance as having a best audio quality of the at least some of the plurality of candidate emotionally enhanced utterances;
outputting, by the software code executed by the processing hardware, an emotionally attributed dialogue data providing the validated emotionally enhanced utterance for use by the digital character in the conversation; and
synthesizing, by the software code executed by the processing hardware using the speech synthesizer and the emotionally attributed dialogue data, the validated emotionally enhanced utterance to generate a synthesized speech for utterance by the digital character.
13. The method of claim 12 , further comprising:
the digital character uttering the synthesized speech.
14. The method of claim 12 , wherein the plurality of structural levels of the utterance comprise at least two of token level, a phrase level, or an entire utterance level.
15. The method of claim 12 , wherein the predictive model comprises the trained ML model, and wherein the trained ML model comprises a transformer-based token insertion ML model.
16. The method of claim 12 , wherein the one or more emotional attributions identify one or more of a prosodic variation, or a word rate.
17. The method of claim 12 , wherein the conversation is between the digital character and a user of the system, the method further comprising:
obtaining, by the software code executed by the processing hardware, a user profile of the user, the user profile including a user history of the user; and
analyzing the emotionality of the utterance further using the user profile.
18. The method of claim 12 , wherein the digital character is associated with a character persona, the method further comprising:
obtaining, by the software code executed by the processing hardware, the character persona; and
analyzing the emotionality of the utterance further using the character persona.
19. The method of claim 12 , further comprising:
determining, by the software code executed by the processing hardware and using another predictive model of the software code, a quality score for each of the plurality of candidate emotionally enhanced utterances to provide a plurality of quality scores corresponding respectively to the plurality of candidate emotionally enhanced utterances; and
identifying, by the software code executed by the processing hardware based on the plurality of quality scores, the at least some of the plurality of candidate emotionally enhanced utterances for the audio validation.
20. The method of claim 12 , wherein the dialogue data and the emotionally attributed dialogue data comprise text.
21. The method of claim 12 , wherein the validated emotionally enhanced utterance further include a non-verbal filler.