Controllable, natural paralinguistics for text to speech synthesis
A speech recognition module receives training data of speech and creates a representation for individual words, non-words, phonemes, and any combination. A set of speech processing detectors analyze the training data of speech from humans communicating. The set of speech processing detectors detect speech parameters that are indicative of paralinguistic effects on top of enunciated words, phonemes, and non-words in the audio stream. One or more machine learning models undergo supervised machine learning on their neural network to train on how to associate one or more mark-up markers with a textual representation, for each individual word, individual non-word, individual phoneme, and any combinations of these, that was enunciated with a particular paralinguistic effect. Each mark-up marker can correspond to its own paralinguistic effect.
1. An apparatus, comprising:
one or more machine learning models trained to examine audio data that includes at least one of i) a word, ii) a phoneme, and iii) a non-word, in a speech communication, that is annotated with one or more mark-up markers and guide a generation of a textual representation to cause a different enunciation from a plain enunciation that would occur without the one or more mark-up markers in order to convey an additional intended meaning of an enunciated i) word, ii) phoneme, and/or iii) non-word, themselves conveyed by a pairing of the i) word, ii) phoneme, or iii) non-word with the different enunciation, where the one or more machine learning models were trained with training data of humans speaking with paralinguistic effects,
where a speech generation module is configured to receive the textual representation generated to guide the speech generation module so that the speech generation module is configured to create speech of the i) word, ii) phoneme, or iii) non-word in the different enunciation with a particular paralinguistic effect in a manner that better conveys the additional intended meaning of the enunciated i) word, ii) phoneme, and/or iii) non-word, themselves; and
where any software portions of the speech generation module and the one or more machine learning models are stored in one or more non-transitory storage mediums in an executable state by one or more processors.
2. The apparatus of claim 1 , further comprising:
a table is configured to map a set of mark-up markers, where each different mark-up marker is mapped to its corresponding particular paralinguistic effect on the enunciated i) word, ii) phoneme, and/or iii) non-word; and
where the one or more machine learning models are configured to cooperate with the table to produce the particular paralinguistic effect changes in the enunciation of the enunciated i) word, ii) phoneme, and/or iii) non-word when conveying the particular paralinguistic effect.
3. The apparatus of claim 1 , further comprising:
a natural language generator module is configured to generate the textual representation of the i) word, ii) phoneme, and/or iii) non-word;
a table of mark-up markers, where each mark-up marker corresponds its own paralinguistic effect; and
where the table is referenced by the natural language generator module to mark-up the textual representation of the i) word, ii) phoneme, or iii) non-word, with the one or more mark-up markers to guide the speech generation module on how to enunciate the i) word, ii) phoneme, or iii) non-word, with the particular paralinguistic effect to cause the different enunciation from the plain enunciation of that i) word, ii) phoneme, or iii) non-word.
4. The apparatus of claim 3 , where the different enunciation differs in one or more speech parameters from the plain enunciation of that i) word, ii) phoneme, or iii) non-word by a threshold amount.
5. The apparatus of claim 1 , where the one or more machine learning models are trained with supervised machine learning on how to identify the particular paralinguistic effects and corresponding mark-up markers, as well as then to understand the additional-intended meaning being conveyed with each particular paralinguistic effect.
6. An apparatus, comprising:
a set of speech processing detectors are configured to analyze training data of an audio stream of speech from humans communicating, where the set of speech processing detectors detect speech parameters that are indicative of one or more paralinguistic effects on top of enunciated words, phonemes, and non-words in the audio stream;
a speech recognition module is configured to receive the training data of the audio stream of speech and create a textual representation for i) individual words, ii) individual non-words, iii) individual phonemes, and any combinations of these, in the training data of the audio stream of speech;
one or more machine learning models that employ neural networks are configured to undergo supervised machine learning to train on how to associate one or more mark-up markers with the textual representation for each i) individual word, ii) individual non-word, and/or iii) individual phoneme that was enunciated with a particular paralinguistic effect; and
where the set of speech processing detectors, the speech recognition module, and the one or more machine learning models are configured to cooperate to automate a labeling of the training data and a pre-deployment training of the one or more machine learning models trained on the one or more paralinguistic effects, where a first mark-up marker corresponds to a first paralinguistic effect and a second mark-up marker corresponds to a second paralinguistic effect, where the first mark-up marker with the first paralinguistic effect is configured to change a meaning conveyed by the i) individual word, ii) individual non-word, and/or iii) individual phoneme that was enunciated with the first paralinguistic effect compared to an enunciation of the i) individual word, ii) individual non-word, and/or iii) individual phoneme with the second paralinguistic effect.
7. The apparatus of claim 6 , where the one or more machine learning models are trained on changes in prosody including prosodic patterns as well as changes of individual speech parameters to associate and understand the changes on the i) individual words, ii) individual non-words, and/or iii) individual phonemes to particular paralinguistic effects detected by the set of speech processing detectors in the training data of the audio stream of speech.
8. The apparatus of claim 6 , further comprising:
where the one or more machine learning models are trained to look for patterns, and
where the set of speech processing detectors are configured to analyze the training data to assist putting annotations onto the training data so that the one or more machine learning models can correlate, at least 1) a first particular paralinguistic effect to at least one of i) an intended additional intended meaning and ii) a more comprehensible of way of conveying information with the first particular paralinguistic effect.
9. The apparatus of claim 6 , where the one or more machine learning models are trained to learn how to identify each different paralinguistic effect and its corresponding markup marker as well as then how to produce corresponding waveforms when conveying the first paralinguistic effect and the second paralinguistic effect.
10. The apparatus of claim 6 , further comprising:
a representation marker module is configured to annotate the textual representation of the i) individual words, ii) individual non-words, and/or iii) individual phonemes with the first mark-up marker corresponding to the first paralinguistic effect that indicate that particular paralinguistic effect.
11. A method for a conversational engagement platform, comprising:
configuring one or more machine learning models trained to examine audio data that includes at least one of i) a word, ii) a phoneme, and iii) a non-word, in a speech communication, that is annotated with one or more mark-up markers to guide a generation of a textual representation to cause a different enunciation from a plain enunciation that would occur without the one or more mark-up markers in order to convey an additional intended meaning of an enunciated i) word, ii) phoneme, and/or iii) non-word, themselves conveyed by a pairing of the i) word, ii) phoneme, or iii) non-word with the different enunciation compared to the plain enunciation, where the one or more machine learning models were trained with training data of humans speaking with paralinguistic effects,
configuring a speech generation module to receive the textual representation generated to guide the speech generation module to generate speech in the different enunciation of the i) word, ii) phoneme, or iii) non-word with a particular paralinguistic effect in a manner that better conveys the additional intended meaning of the enunciated i) word, ii) phoneme, and/or iii) non-word, themselves conveyed by a pairing of the i) word, ii) phoneme, or iii) non-word with the different enunciation; and
where any software portions of the speech generation module and the one or more machine learning models are stored in one or more non-transitory storage mediums in an executable state by one or more processors.
12. The method for the conversational engagement platform of claim 11 , further comprising:
configuring a table to map a set of mark-up markers, where each different mark-up marker is mapped to its corresponding particular paralinguistic effect on the enunciated i) word, ii) phoneme, and/or iii) non-word; and
configuring the one or more machine learning models to cooperate with the table to produce the particular paralinguistic effect changes in the enunciation of the enunciated i) word, ii) phoneme, and/or iii) non-word when conveying the particular paralinguistic effect.
13. The method for the conversational engagement platform of claim 11 , further comprising:
configuring a natural language generator module to generate the textual representation of the i) word, ii) phoneme, and/or iii) non-word;
configuring a table of mark-up markers, where each mark-up marker corresponds its own paralinguistic effect; and
configuring the table to be referenced by the natural language generator module to mark-up the textual representation of the i) word, ii) phoneme, or iii) non-word, with the one or more mark-up markers to guide the speech generation module on how to enunciate the i) word, ii) phoneme, or iii) non-word with the particular paralinguistic effect to cause the different enunciation from the plain enunciation of that i) word, ii) phoneme, or iii) non-word.
14. The method for the conversational engagement platform of claim 13 , where the different enunciation differs in one or more speech parameters from the plain enunciation of that i) word, ii) phoneme, or iii) non-word by a threshold amount.
15. The method for the conversational engagement platform of claim 11 , where the one or more machine learning models are trained with supervised machine learning on how to identify the particular paralinguistic effects and corresponding mark-up markers, as well as then to understand the additional intended meaning being conveyed with each particular paralinguistic effect.
16. A method for a conversational engagement platform, comprising:
configuring a set of speech processing detectors to analyze training data of an audio stream of speech from humans communicating, where the set of speech processing detectors detect speech parameters that are indicative of one or more paralinguistic effects on top of enunciated words, phonemes, and non-words in the audio stream;
configuring a speech recognition module to receive the training data of the audio stream of speech and create a textual representation for i) individual words, ii) individual non-words, and/or iii) individual phonemes, and any combinations of these, in the training data of the audio stream of speech;
training one or more machine learning models that employ neural networks with supervised machine learning to train on how to associate one or more mark-up markers with the textual representation for each i) individual word, ii) individual non-word, and/or iii) individual phoneme that was enunciated with a particular paralinguistic effect;
configuring the set of speech processing detectors, where the speech recognition module, and the one or more machine learning models cooperate to automate a labeling of the training data and a pre-deployment training of the one or more machine learning models trained on the one or more paralinguistic effects, where a first mark-up marker corresponds to a first paralinguistic effect and a second mark-up marker corresponds to a second paralinguistic effect; and
configuring the first mark-up marker with the first paralinguistic effect to change a meaning conveyed by the i) individual word, ii) individual non-word, and/or iii) individual phoneme that was enunciated with the first paralinguistic effect compared to an enunciation of the i) individual word, ii) individual non-word, and/or iii) individual phoneme with the second paralinguistic effect.
17. The method for the conversational engagement platform of claim 16 , where the one or more machine learning models are trained on changes in prosody including prosodic patterns as well as changes of individual speech parameters to associate and understand the changes on the i) individual words, ii) individual non-words, and iii) individual phonemes to particular paralinguistic effects detected by the set of speech processing detectors in the training data of the audio stream of speech.
18. The method for the conversational engagement platform of claim 16 , further comprising:
configuring the one or more machine learning models trained to look for patterns, and
analyzing the training data by with the set of speech processing detectors to put annotations onto the training data so that the one or more machine learning models can correlate, at least 1) a first particular paralinguistic effect to at least one of i) an intended additional intended meaning and ii) a more comprehensible of way of conveying information with the first particular paralinguistic effect.
19. The method of claim 16 , further comprising:
configuring the one or more machine learning models to train on how to learn to identify each different paralinguistic effect and its corresponding markup marker as well as then how to produce corresponding waveforms when conveying the first paralinguistic effect and the second paralinguistic effect.
20. The method of claim 16 , further comprising:
configuring a representation marker module to annotate the textual representation of the i) individual words, ii) individual non-words, and/or iii) individual phonemes with the first mark-up marker corresponding to the first paralinguistic effect that indicate that particular paralinguistic effect.