Stylizing text-to-speech (TTS) voice response for assistant systems
In one embodiment, a method includes receiving a voice input from a user and determining a first style of the voice input, based on first features extracted from the voice input. A second style for a voice response having second features may then be determined based on the first style. Finally, the voice response may be generated based on the second features of the second style, and this voice response may be provided in response to the voice input.
1. A method comprising:
receiving, via an assistant application, a voice input from a user;
generating a first style embedding corresponding to the voice input, wherein the first style embedding is generated based on a prosody of the voice input, one or more audio features of the voice input, and one or more token representations of the voice input;
generating a text response corresponding to the voice input;
generating a second style embedding corresponding to the text response;
generating, an output audio waveform of the text response, wherein the output audio waveform is generated based on the first and second style embeddings and one or more first features extracted from the voice input, wherein the output audio waveform comprises one or more second features; and
providing, via the assistant application, the output audio waveform in response to the voice input.
2. The method of claim 1 , further comprising:
extracting the one or more first features from the voice input.
3. The method of claim 1 , wherein the voice input is in a first style and the output audio waveform is in a second style, and wherein the first style and the second style are the same style.
4. The method of claim 1 , wherein the voice input is in a first style and the output audio waveform is in a second style, and wherein the first style and the second style are different styles.
5. The method of claim 1 , further comprising:
automatically transcribing the voice input into an input text.
6. The method of claim 5 , wherein the first style embedding is further based on the input text.
7. The method of claim 1 , wherein the output audio waveform is further based on environmental features associated with the user.
8. The method of claim 1 , wherein the output audio waveform is further based on a time of receipt associated with the voice input.
9. The method of claim 1 , further comprising:
weighting a plurality of potential styles based on the first and second style embeddings;
selecting a style for the output audio waveform based on the weights associated with the plurality of potential styles.
10. The method of claim 1 , wherein the one or more audio features of the voice input comprise one or more of volume, emphasis, pronunciation, accent, or frequency of the voice input.
11. The method of claim 1 , wherein generating the output audio waveform is further based on one or more characteristics of the user.
12. The method of claim 1 , further comprising:
customizing the output audio waveform based on one or more characteristics of the user.
13. The method of claim 12 , wherein the one or more characteristics comprise one or more of demographics of the user or preferences of the user.
14. The method of claim 1 , further comprising:
determining, by a machine-learning model, a first style of voice input.
15. The method of claim 1 , wherein generating the output audio waveform comprises synthesizing the output audio waveform using a text-to-speech (TTS) pipeline comprising a prosody model and an acoustic model.
16. The method of claim 15 , wherein the prosody model determines a duration and a frequency of the output audio waveform.
17. The method of claim 16 , wherein generating the output audio waveform comprises:
determining a first style of the voice input based on the prosody of the voice input and the one or more audio features of the voice input;
selecting a second style for the output audio waveform based on the first style, wherein the second style comprises the one or more second features;
accessing, by the prosody model, an output style embedding corresponding to the second style; and
setting one or more of the duration or the frequency of the output audio waveform based on the output style embedding to generate response prosody features of the output audio waveform.
18. The method of claim 17 , wherein the output style embedding is determined based on the first style embedding corresponding to the voice input.
19. The method of claim 1 , further comprising:
generating a default output audio waveform of the text response, wherein the default output audio waveform has a default style comprising one or more default features,
wherein generating the output audio waveform comprises modifying one or more of the default features of the default output audio waveform based on one or more of the second features.
20. A system comprising one or more processors and a memory coupled to the processors, the memory comprising instructions that, when executed by the processors, configure the processors to:
receive, via an assistant application, a voice input from a user;
generate a first style embedding corresponding to the voice input, wherein the first style embedding is generated based on a prosody of the voice input, one or more audio features of the voice input, and one or more token representations of the voice input;
generate a text response corresponding to the voice input;
generate a second style embedding corresponding to the text response;
generate, an output audio waveform of the text response, wherein the output audio waveform is generated based on the first and second style embeddings and one or more first features extracted from the voice input, wherein the output audio waveform comprises one or more second features; and
provide, via the assistant application, the output audio waveform in response to the voice input.
21. One or more computer-readable non-transitory storage media embodying software that is configured, when executed by a processor, to:
receive, via an assistant application, a voice input from a user;
generate a first style embedding corresponding to the voice input, wherein the first style embedding is generated based on a prosody of the voice input, one or more audio features of the voice input, and one or more token representations of the voice input;
generate a text response corresponding to the voice input;
generate a second style embedding corresponding to the text response;
generate, an output audio waveform of the text response, wherein the output audio waveform is generated based on the first and second style embeddings and one or more first features extracted from the voice input, wherein the output audio waveform comprises one or more second features; and
provide, via the assistant application, the output audio waveform in response to the voice input.