System and method for cross-speaker style transfer in text-to-speech and training data generation

1. A method implemented by a computing system for generating a spectrogram for a target speaker in a prosody style of a source speaker and for training a machine learning model based on the spectrogram, the method comprising:

receiving electronic content comprising source speaker data from the source speaker;

converting a waveform of the source speaker data to phonetic posterior gram (PPG) data, wherein said converting includes aligning the waveform to the PPG data;

generating a spectrogram based on the PPG data, wherein the spectrogram is characterized by a determined prosody style of the source speaker; and

generating speech data from arbitrary text using a neural text-to-speech (TTS) model that is trained with the generated spectrogram, the speech data being characterized by the prosody style of the source speaker.

2. The method of claim 1 , wherein the waveform is aligned to the PPG data at a granularity that is narrower than a phoneme-based granularity.

3. The method of claim 1 , wherein the waveform is aligned to the PPG data at a frame-based granularity.

4. The method of claim 3 , wherein the frame-based granularity is based on a plurality of frames, and wherein each frame in said plurality of frames is a selected number of milliseconds.

5. The method of claim 1 , wherein one or more additional prosody features are extracted from the source speaker data, and

wherein the one or more additional prosody features comprise one or more of the following: pitch or energy.

6. The method of claim 5 , wherein the one or more additional prosody features extracted from the source speaker data includes said energy, the energy being measured in terms of a volume of the source speaker data.

7. The method of claim 5 , wherein the one or more additional prosody features are extracted from the waveform of the source speaker data at a frame-based granularity.

8. The method of claim 1 , wherein the method includes the computing system defining the prosody style of the source speaker.

9. The method of claim 8 , wherein the prosody style comprises one or more of the following: a newscaster style, a story-telling style, a serious style, a casual style, a customer service style, or an emotion-based style.

10. The method of claim 9 , wherein the emotion-based style is detected by the computing system to be at least one of: a happy emotion, a sad emotion, an angry emotion, an excited emotion, or an embarrassed emotion.

11. A method implemented by a computing system for training a voice conversion machine learning model within a voice conversion module to generate a spectrogram for a target speaker with a new prosody style of a source speaker, the method comprising:

training a phonetic posterior gram-(PPG)-to-spectrogram component of the voice conversion machine learning model to convert PPG data into spectrogram data;

refining the PPG-to-spectrogram component with target speaker data from a target speaker having a particular prosody style, said refining being performed by adapting the PPG-to-spectrogram component to convert PPG data to spectrogram data having the particular prosody style;

receiving electronic content comprising new PPG data converted from a waveform of source speaker data, the new PPG data being aligned to the waveform of the source speaker data; and

applying the source speaker data to the voice conversion machine learning model, wherein the refined PPG-to-spectrogram component is configured to generate a spectrogram having the new prosody style of the source speaker rather than the particular prosody style of the target speaker.

12. The method of claim 11 , wherein the new PPG data is aligned to the waveform of the source speaker data at a granularity that is narrower than a phoneme-based granularity.

13. The method of claim 11 , wherein the new PPG data is aligned to the waveform of the source speaker data at a frame-based granularity.

14. The method of claim 13 , wherein the frame-based granularity is based on a plurality of frames, each frame comprising approximately a particular number of milliseconds.

15. The method of claim 11 , wherein the method further includes:

extracting one or more prosody features from the waveform of the source speaker data, said one or more prosody features comprising at least one of: a pitch contour, an energy contour, a speaking duration, or a speaking rate.

16. The method of claim 15 , wherein the refined PPG-to-spectrogram component is further configured based on at least one of: the pitch contour, the energy contour, the speaking duration, or the speaking rate.

17. The method of claim 15 , wherein the pitch contour and/or the energy contour are extracted from the waveform of the source speaker data at a frame-based granularity.

18. A method implemented by a computing system for generating training data for training a neural text-to-speech (TTS) model configured to generate speech data from arbitrary text, the method comprising:

receiving electronic content comprising source speaker data from a source speaker;

converting a waveform of the source speaker data to phonetic posterior gram (PPG) data, wherein said converting includes aligning the waveform to the PPG data;

applying the PPG data to a pre-trained PPG-to-spectrogram component of a voice conversion module, the pre-trained PPG-to-spectrogram component configured to generate a spectrogram having a new prosody style of the source speaker rather than a particular prosody style of a target speaker;

generating training data configured to train a neural TTS model, the training data comprising a plurality of spectrograms characterized by the new prosody style of the source speaker; and

generating speech data from arbitrary text using the neural TTS model that is trained on the generated training data, said generating of the speech data is performed using cross-speaker style transfer, and the speech data is characterized by the new prosody style of the source speaker.

19. The method of claim 18 , wherein the method includes the computing system defining the new prosody style of the source speaker.

20. The method of claim 19 , said method further comprising:

receiving electronic content comprising the arbitrary text;

applying the arbitrary text as input to the trained neural TTS model; and

generating output comprising the speech data, which is based on the arbitrary text, wherein the speech data is characterized by the prosody style of the source speaker and a voice timbre of the target speaker.