Computing system for unsupervised emotional text to speech training
A text to speech (TTS) model is trained based on training data including text samples. The text samples are provided to a text embedding model for outputting text embeddings for the text samples. The text embeddings are clustered into several clusters of text embeddings. The several clusters are representative of variations in emotion. The TTS model is then trained based upon the several clusters of text embeddings. Upon being trained, the TTS model is configured to receive text input and output a spoken utterance that corresponds to the text input. The TTS model is configured to output the spoken utterance with emotion. The emotion is based upon the text input and the training of the TTS model.
1 . A system, comprising:
a processor; and
memory storing instructions that, when executed by the processor, cause the processor to perform acts comprising:
receiving training data for training a text to speech (TTS) model, wherein the training data includes text that comprises text samples, wherein the training data does not include emotional labels indicative of one or more emotions expressed in the text; and
training the TTS model based upon the training data, wherein training the TTS model comprises:
providing the text samples to a text embedding model, wherein the text embedding model outputs text embeddings for the text samples wherein the text embeddings comprise one or more emotions encoded in the text embeddings by the text embedding model;
clustering the text embeddings into several clusters of text embeddings, wherein the several clusters are representative of variations in emotion encoded in the textual embeddings; and
training the TTS model based upon the several clusters of text embeddings, wherein the TTS model, upon being trained, is configured to receive text input and output a spoken utterance that corresponds to the text input, wherein the TTS model is configured to output the spoken utterance with an emotion, and further wherein the emotion is based upon the text input and the training of the text to speech model.
2 . The system of claim 1 , wherein the text embeddings are clustered into the several clusters of text embeddings by way of k-means clustering.
3 . The system of claim 1 , wherein the training data is does not include labels that identify emotions, and further wherein training the TTS model is accomplished by way of unsupervised learning.
4 . The system of claim 1 , wherein the several clusters are 16 clusters.
5 . The system of claim 1 , the acts further comprising:
identifying cluster heads for the several clusters, where the cluster heads are centroids of the several clusters.
6 . The system of claim 1 , the acts further comprising:
subsequent to training the TTS model, receiving the text input;
providing the text input to the TTS model;
generating, by the TTS model, the spoken utterance based upon the text input; and
causing a speaker of a client computing device to audibly output the spoken utterance.
7 . The system of claim 1 , wherein the training data further comprises audio samples that correspond to the text samples, and wherein training the TTS model further comprises:
providing the audio samples to an audio embedding model, wherein the audio embedding model outputs audio embeddings, and further wherein training the TTS model is based upon the several clusters of text embeddings and the audio embeddings.
8 . The system of claim 7 , wherein the audio embedding model is a Global Style Token model.
9 . The system of claim 7 , wherein training the TTS model based upon the training data further comprises:
projecting a text embedding for a text sample in the training data into audio embedding space to form a projected embedding;
projecting embeddings of cluster heads for the clusters into the audio embedding space to form projected cluster head embeddings; and
selecting, by an attention mechanism, a projected cluster head embedding in the projected cluster head embeddings based upon an audio embedding in the audio embeddings that corresponds to the text sample, wherein the TTS model is trained based upon the selected projected cluster head embedding.
10 . The system of claim 9 , wherein training the TTS model further comprises:
identifying a cluster head embedding based upon the selected projected cluster head embedding, wherein the TTS model is trained based upon the identified cluster head embedding.
11 . A method for training a text to speech model (TTS) that, the method comprising:
obtaining training data for training the TTS model, wherein the training data comprises text samples and audio samples, wherein the training data does not include emotional labels indicative of one or more emotions expressed in the text samples and the audio samples;
providing the text samples to a text embedding model, wherein the text embedding model outputs text embeddings for the text samples wherein the text embeddings comprise one or more emotions encoded in the text embeddings by the text embedding model;
clustering the text embeddings into several clusters of text embeddings, wherein the several clusters are representative of variations in emotion encoded in the textual embeddings; and
training the TTS model based upon the several clusters of text embeddings, wherein the TTS model, upon being trained, is configured to receive text input and output a spoken utterance that corresponds to the text input, wherein the TTS model is configured to output the spoken utterance with an emotion, and further wherein the emotion is based upon the text input and the training of the text to speech model.
12 . The method of claim 11 , wherein training the TTS model comprises:
converting the audio samples into spectrograms; and
generating audio embeddings for the audio samples based upon the spectrograms, wherein a Global Style Token is employed to generate the audio embeddings, and further wherein the TTS model is trained based upon the audio embeddings.
13 . The method of claim 11 , wherein training the TTS model comprises:
generating text embeddings for the text samples, wherein the text embeddings are vectors of uniform length that comprise one or more emotions encoded in the text embeddings;
clustering the text embeddings into a predefined number of clusters, where different clusters correspond to different emotions encoded in the textual embeddings;
projecting the text embeddings for the text samples into audio embedding space to form projected embeddings, wherein the TTS model is trained based upon the projected embeddings.
14 . The method of claim 13 , wherein training the TTS model comprises:
determining cluster heads for each of the predefined number of clusters; and
projecting the cluster heads into the audio embedding space to form projected cluster head embeddings, wherein the TTS model is trained based upon the projected cluster head embeddings.
15 . The method of claim 14 , wherein a projection module that comprises a 2-layer nonlinear feed forward network projects the text embeddings and the cluster head embeddings into the audio embedding space.
16 . The method of claim 14 , further comprising:
using an attention mechanism to select a projected cluster head embedding from amongst the projected cluster head embeddings, wherein the attention mechanism selects the projected cluster head embedding based upon an audio embedding for an audio sample, and further wherein the TTS model is trained based upon the selected projected cluster head embedding.
17 . The method of claim 16 , wherein the TTS model is further trained based upon a difference between the audio embedding for the audio sample and a projected text embedding for a text sample that corresponds to the audio sample.
18 . A non-transitory computer-readable storage medium comprising instructions that, when executed by a processor of a computing device, cause the processor to perform acts comprising:
receiving training data for training a text to speech (TTS) model, wherein the training data includes text that comprises text samples, wherein the training data does not include emotional labels indicative of one or more emotions expressed in the text; and
training the TTS model based upon the training data, wherein training the TTS model comprises:
providing the text samples to a text embedding model, wherein the text embedding model outputs text embeddings for the text samples, wherein the text embeddings comprise one or more emotions encoded in the text embeddings by the text embedding model;
clustering the text embeddings into several clusters of text embeddings, wherein the several clusters are representative of variations in emotion encoded in the textual embeddings; and
training the TTS model based upon the several clusters of text embeddings, wherein the TTS model, upon being trained, is configured to receive text input and output a spoken utterance that corresponds to the text input, wherein the TTS model is configured to output the spoken utterance with an emotion, and further wherein the emotion is based upon the text input and the training of the text to speech model.
19 . The non-transitory computer-readable storage medium of claim 18 , wherein the training data further comprises audio samples that correspond to the text samples, and wherein training the TTS model further comprises:
providing the audio samples to an audio embedding model, wherein the audio embedding model outputs audio embeddings, and wherein training the TTS model is based upon the several clusters of text embeddings and the audio embeddings.
20 . The non-transitory computer-readable storage medium of claim 19 , wherein the audio embedding model is a Global Style Token.