Systems and methods for generating synthesized speech responses to voice inputs by training a neural network model based on the voice input prosodic metrics and training voice inputs

1. A computer-implemented method for training a neural network model to provide information used to provide a synthesized speech response to a voice input, the method comprising:

receiving a plurality of voice inputs, each associated with at least one respective voice input prosodic metric;

extracting training data from the plurality of voice inputs and the associated voice input prosodic metrics;

processing the extracted training data for input into the neural network model;

receiving a plurality of responses, each associated with at least one respective response prosodic metric;

training the neural network model based on the extracted training data including the processed plurality of voice inputs, the plurality of responses, the associated and processed voice input prosodic metrics, and the associated response prosodic metrics; and

outputting, with the trained neural network model, the information used to generate the synthesized speech response to the voice input.

2. The method of claim 1 , wherein:

the at least one voice input prosodic metric is selected from the group comprising pitch, note, duration, prominence, timbre, rate, and rhythm;

the at least one response prosodic metric is selected from the group comprising pitch, note, duration, prominence, timbre, rate, and rhythm; and

the processing the extracted training data for input into the neural network model includes for at least a portion of each of the plurality of voice inputs an association with at least one from the group comprising a pitch indicator, a pitch for each frame, a pitch interval, a note, a speaking speed, a number of syllables, an age of an entity associated with the voice input, and a sentiment.

3. The method of claim 1 , wherein each respective voice input of the plurality of voice inputs comprises a first plurality of words, and wherein each respective response of the plurality of responses comprises a second plurality of words, further comprising, for each respective voice input and for each respective response:

receiving one or more first word transition metrics among words of the first plurality of words;

receiving one or more second word transition metrics among words of the second plurality of words; and

training the neural network model further based on each first word transition and each second word transition.

4. The method of claim 1 , wherein:

each respective voice input of the plurality of voice inputs comprises a first plurality of words;

the at least one respective voice input prosodic metric is associated with one or more words of the first plurality of words;

each respective response of the plurality of responses comprises a second plurality of words; and

the at least one respective response prosodic metric is associated with one or more words of the second plurality of words.

5. The method of claim 1 , wherein the plurality of voice inputs and the plurality of responses are associated in a database, the method further comprising retrieving the plurality of voice inputs and the plurality of responses from the database.

6. The method of claim 1 , further comprising:

receiving a first emotion metric for each respective voice input;

receiving a second emotion metric for each respective response corresponding to the respective voice input; and

training the neural network model further based on each first emotion metric and each second emotion metric.

7. The method of claim 1 , further comprising:

receiving user profile information selected from the group comprising user voice input history, user language, user characteristics, user location, user preferences, and metadata tags associated with the user; and

training the neural network model further based on the user profile information.

8. The method of claim 1 , further comprising:

receiving respective interpolation metrics among a transition of words of each voice input of the plurality of voice inputs; and

training the neural network model further based on each respective interpolation metric.

9. The method of claim 1 , wherein each voice input of the plurality of voice inputs is linked with a respective set of responses of the plurality of responses.

10. The method of claim 1 , wherein the information used to generate the synthesized speech response comprises prosodic characteristics.

11. A system for training a neural network model to provide information used to provide a synthesized speech response to a voice input, the system comprising:

control circuitry configured to:

receive a plurality of voice inputs, each associated with at least one respective voice input prosodic metric;

extract training data from the plurality of voice inputs and the associated voice input prosodic metrics;

process the extracted training data for input into the neural network model;

receive a plurality of responses, each associated with at least one respective response prosodic metric;

train the neural network model based on the extracted training data including the processed plurality of voice inputs, the plurality of responses, the associated and processed voice input prosodic metrics, and the associated response prosodic metrics; and

output, with the trained neural network model, the information used to generate the synthesized speech response to the voice input; and

a storage device for storing the information.

12. The system of claim 11 , wherein:

the at least one voice input prosodic metric is selected from the group comprising pitch, note, duration, prominence, timbre, rate, and rhythm;

the at least one response prosodic metric is selected from the group comprising pitch, note, duration, prominence, timbre, rate, and rhythm; and

the process of the extracted training data for input into the neural network model includes for at least a portion of each of the plurality of voice inputs an association with at least one from the group comprising a pitch indicator, a pitch for each frame, a pitch interval, a note, a speaking speed, a number of syllables, an age of an entity associated with the voice input, and a sentiment.

13. The system of claim 11 , wherein each respective voice input of the plurality of voice inputs comprises a first plurality of words, and wherein each respective response of the plurality of responses comprises a second plurality of words, and wherein the control circuitry is further configured to, for each respective voice input and for each respective response:

receive one or more first word transition metrics among words of the first plurality of words;

receive one or more second word transition metrics among words of the second plurality of words; and

train the neural network model further based on each first word transition and each second word transition.

14. The system of claim 11 , wherein:

each respective voice input of the plurality of voice inputs comprises a first plurality of words;

the at least one respective voice input prosodic metric is associated with one or more words of the first plurality of words;

each respective response of the plurality of responses comprises a second plurality of words; and

the at least one respective response prosodic metric is associated with one or more words of the second plurality of words.

15. The system of claim 11 , wherein the plurality of voice inputs and the plurality of responses are associated in a database, and wherein the control circuitry is further configured to retrieve the plurality of voice inputs and the plurality of responses from the database.

16. The system of claim 11 , wherein the control circuitry is further configured to:

receive a first emotion metric for each respective voice input;

receive a second emotion metric for each respective response corresponding to the respective voice input; and

train the neural network model further based on each first emotion metric and each second emotion metric.

17. The system of claim 11 , wherein the control circuitry is further configured to:

receive user profile information selected from the group comprising user voice input history, user language, user characteristics, user location, user preferences, and metadata tags associated with the user; and

train the neural network model further based on the user profile information.

18. The system of claim 11 , wherein the control circuitry is further configured to:

receive respective interpolation metrics among a transition of words of each voice input of the plurality of voice inputs; and

train the neural network model further based on each respective interpolation metric.

19. The system of claim 11 , wherein each voice input of the plurality of voice inputs is linked with a respective set of responses of the plurality of responses.

20. The system of claim 11 , wherein the information used to generate the synthesized speech response comprises prosodic characteristics.