Method for synthesized speech generation using emotion information correction and apparatus
A method includes generating first synthesized speech by using text and a first emotion vector configured for the text, extracting a second emotion vector included in the first synthesized speech, determining whether correction of the second emotion information vector is needed by comparing a loss value calculated by using the first emotion information vector and the second emotion information vector with a preconfigured threshold, re-performing speech synthesis by using a third emotion information vector generated by correcting the second emotion information vector, and outputting the generated synthesized speech, thereby configuring emotion information of speech in a more effective manner. A speech synthesis apparatus may be associated with an artificial intelligence module, drone (unmanned aerial vehicle, UAV), robot, augmented reality (AR) devices, virtual reality (VR) devices, devices related to 5G services, and the like.
1. A method for generating synthesized speech, the method comprising:
generating first synthesized speech by using text and a first emotion information vector configured for the text;
extracting a second emotion information vector included in the first synthesized speech;
determining whether correction of the second emotion information vector is needed by comparing a loss value calculated by using the first emotion information vector and the second emotion information vector with a preconfigured threshold;
based on the loss value calculated by using the first emotion information vector and the second emotion information vector being less than the preconfigured threshold, outputting the first synthesized speech; and
based on the loss value calculated by using the first emotion information vector and the second emotion information vector exceeding the preconfigured threshold, generating a third emotion information vector by correcting the second emotion information vector based on the first emotion information vector, generating second synthesized speech by using the third emotion information vector, and outputting the second synthesized speech,
wherein a loss value calculated by using the first emotion information vector and an emotion information vector included in the second synthesized speech is less than the preconfigured threshold.
2. The method of claim 1 , wherein the loss value calculated by using the first emotion information vector and the second emotion information vector is a value calculated based on a difference between the first emotion information vector and the second emotion information vector; and
the loss value calculated by using the first emotion information vector and an emotion information vector included in the second synthesized speech is a value calculated based on a difference between the first emotion information vector and an emotion information vector included in the second synthesized speech.
3. The method of claim 2 , wherein the loss value calculated by using the first emotion information vector and an emotion information vector included in the second synthesized speech is 0.
4. The method of claim 1 , wherein the loss value calculated by using the first emotion information vector and the second emotion information vector is a value calculated based on the square of a difference between the first emotion information vector and the second emotion information vector; and
the loss value calculated by using the first emotion information vector and an emotion information vector included in the second synthesized speech is a value calculated based on the square of a difference between the first emotion information vector and an emotion information vector included in the second synthesized speech.
5. The method of claim 1 , wherein the third emotion information vector is generated by using a deep learning model.
6. The method of claim 5 , wherein the deep learning model is a model performing deep learning by using the first emotion information vector, the second emotion information vector, and the third emotion information vector.
7. An apparatus for generating synthesized speech, the apparatus comprising:
an input unit receiving text and a first emotion information vector configured for the text;
an output unit outputting synthesized speech; and
a processor functionally connected to the input unit and the output unit,
wherein the processor is configured to:
generate first synthesized speech by using the text and a first emotion information vector configured for the text;
extract a second emotion information vector included in the first synthesized speech;
determine whether correction of the second emotion information vector is needed by comparing a loss value calculated by using the first emotion information vector and the second emotion information vector with a preconfigured threshold;
based on the loss value calculated by using the first emotion information vector and the second emotion information vector being less than the preconfigured threshold, output the first synthesized speech; and
based on the loss value calculated by using the first emotion information vector and the second emotion information vector exceeding the preconfigured threshold, generate a third emotion information vector by correcting the second emotion information vector based on the first emotion information vector, generate second synthesized speech by using the third emotion information vector, and output the second synthesized speech,
wherein a loss value calculated by using the first emotion information vector and an emotion information vector included in the second synthesized speech is less than the preconfigured threshold, and the synthesized speech is the second synthesized speech.
8. The apparatus of claim 7 , wherein the loss value calculated by using the first emotion information vector and the second emotion information vector is a value calculated based on a difference between the first emotion information vector and the second emotion information vector; and
the loss value calculated by using the first emotion information vector and an emotion information vector included in the second synthesized speech is a value calculated based on a difference between the first emotion information vector and an emotion information vector included in the second synthesized speech.
9. The apparatus of claim 8 , wherein the loss value calculated by using the first emotion information vector and an emotion information vector included in the second synthesized speech is 0.
10. The apparatus of claim 7 , wherein the loss value calculated by using the first emotion information vector and the second emotion information vector is a value calculated based on the square of a difference between the first emotion information vector and the second emotion information vector; and
the loss value calculated by using the first emotion information vector and an emotion information vector included in the second synthesized speech is a value calculated based on the square of a difference between the first emotion information vector and an emotion information vector included in the second synthesized speech.
11. The apparatus of claim 7 , wherein the third emotion information vector is generated by using a deep learning model.
12. The apparatus of claim 11 , wherein the deep learning model is a model performing deep learning by using the first emotion information vector, the second emotion information vector, and the third emotion information vector.
13. An electronic device comprising:
one or more processors;
a memory; and
one or more programs configured to be stored in the memory and to be executed by the one or more processors, the one or more programs including commands for performing the method of claim 1 .