Electronic device for generating mouth shape and method for operating thereof
An electronic device includes at least one processor, and at least one memory storing instructions executable by the at least one processor and operatively connected to the at least one processor, where the at least one processor is configured to acquire voice data to be synthesized with at least one first image, generate a plurality of mouth shape candidates by using the voice data, select a mouth shape candidate among the plurality of mouth shape candidates, generate at least one second image based on the selected mouth shape candidate and at least a portion of each of the at least one first image, and generate at least one third image by applying at least one super-resolution model to the at least one second image.
1. An electronic device comprising:
at least one processor; and
at least one memory storing instructions executable by the at least one processor and operatively connected to the at least one processor,
wherein the instructions, when executed by the at least one processor, cause the electronic device to:
acquire voice data to be synthesized with first images,
generate a plurality of mouth shape candidates for each of the first images by using the voice data,
provide a user interface capable of selecting one of the plurality of mouth shape candidates, wherein the user interface provides a function of outputting a voice corresponding to the voice data while reproducing at least some of the plurality of mouth shape candidates or at least some areas of face areas including each of at least some of the plurality of mouth shape candidates,
based on obtaining an input with respect to the user interface, select a mouth shape candidate among the plurality of mouth shape candidates for each of the first images to obtain a plurality of selected mouth shape candidates corresponding to the first images,
generate second images based on the plurality of selected mouth shape candidates and at least a portion of each of the first images, and
generate third images by applying at least one super-resolution model to the generated second images.
2. The electronic device of claim 1 , wherein the instructions, when executed by the at least one processor, cause the electronic device to generate at least one processed voice data by performing at least one analog processing on the voice data, input the voice data and the at least one processed voice data into a mouth shape generation model, and generate a plurality of output values from the mouth shape generation model as the plurality of mouth shape candidates.
3. The electronic device of claim 2 , wherein the at least one analog processing includes at least one of an increase in an amplitude of the voice data, a decrease in the amplitude of the voice data, an increase of a reproduction speed of the voice data, a decrease in the reproduction speed of the voice data, addition of first noise to the voice data, suppression of second noise from the voice data, separation of a first background sound from the voice data, or addition of a second background sound to the voice data.
4. The electronic device of claim 1 , wherein the instructions, when executed by the at least one processor, cause the electronic device to input the voice data into a plurality of mouth shape generation models and generate a plurality of output values from the plurality of mouth shape generation models as the plurality of mouth shape candidates.
5. The electronic device of claim 1 , wherein the instructions, when executed by the at least one processor, cause the electronic device to provide a user interface capable of selecting a time section in which the voice data is synthesized and identify the first images corresponding to the time section in which the voice data is synthesized, based on a user input made through the user interface.
6. The electronic device of claim 1 , wherein the instructions, when executed by the at least one processor, cause the electronic device to input each of the plurality of mouth shape candidates into an assessment model and select one of the plurality of mouth shape candidates, based on a plurality of scores output from the assessment model.
7. The electronic device of claim 1 , wherein the instructions, when executed by the at least one processor, cause the electronic device to generate a plurality of completely synthesized images by synthesizing at least some of the third images with the first images, respectively.
8. The electronic device of claim 7 , wherein the instructions, when executed by the at least one processor, cause the electronic device to perform segmentation on each of the third images, identify at least one area in each of the third images, based on a result of the segmentation, and generate the plurality of completely synthesized images by synthesizing the at least one area with the first images, respectively.
9. The electronic device of claim 1 , wherein the instructions, when executed by the at least one processor, cause the electronic device to generate the voice data, based on a voice received through a microphone of the electronic device or convert text into the voice data.
10. A method of operating an electronic device, the method comprising:
acquiring voice data to be synthesized with first images,
generating a plurality of mouth shape candidates for each of the first images by using the voice data,
providing a user interface capable of selecting one of the plurality of mouth shape candidates, wherein the user interface provides a function of outputting a voice corresponding to the voice data while reproducing at least some of the plurality of mouth shape candidates or at least some areas of face areas including each of at least some of the plurality of mouth shape candidates,
based on obtaining an input with respect to the user interface, selecting a mouth shape candidate among the plurality of mouth shape candidates for each of the first images to obtain a plurality of selected mouth shape candidates corresponding to the first images,
generating second images based on the plurality of selected mouth shape candidates and at least a portion of each of the first images, and
generating third images by applying at least one super-resolution model to the second images.
11. The method of claim 10 , wherein the generating of the plurality of mouth shape candidates comprises:
generating at least one processed voice data by performing at least one analog processing on the voice data,
inputting the voice data and the at least one processed voice data into a mouth shape generation model, and
generating a plurality of output values from the mouth shape generation model as the plurality of mouth shape candidates.
12. The method of claim 11 , wherein the analog processing comprises at least one of an increase in an amplitude of the voice data, a decrease in the amplitude of the voice data, an increase of a reproduction speed of the voice data, a decrease in the reproduction speed of the voice data, addition of first noise to the voice data, suppression of second noise from the voice data, separation of a first background sound from the voice data, or addition of a second background sound to the voice data.
13. The method of claim 10 , wherein the generating of the plurality of mouth shape candidates comprises inputting the voice data into a plurality of mouth shape generation models and generating a plurality of output values from the plurality of mouth shape generation models as the plurality of mouth shape candidates.
14. A non-transitory computer-readable storage medium storing at least one instruction, the at least one instruction causing at least one processor to:
acquire voice data to be synthesized with first images,
generate a plurality of mouth shape candidates for each of the first images by using the voice data,
provide a user interface capable of selecting one of the plurality of mouth shape candidates, wherein the user interface provides a function of outputting a voice corresponding to the voice data while reproducing at least some of the plurality of mouth shape candidates or at least some areas of face areas including each of at least some of the plurality of mouth shape candidates,
based on obtaining an input with respect to the user interface, select a mouth shape candidate among the plurality of mouth shape candidates for each of the first images to obtain a plurality of selected mouth shape candidates corresponding to the first images,
generate second images based on the plurality of selected mouth shape candidates and at least a portion of each of the first images, and
generate third images by applying at least one super-resolution model to the second images.
15. An electronic device comprising:
at least one processor; and
at least one memory storing instructions executable by the at least one processor and operatively connected to the at least one processor,
wherein the instructions, when executed by the at least one processor, cause the electronic device to:
acquire voice data to be synthesized with at least one first image,
generate a plurality of mouth shape candidates by using the voice data,
select a mouth shape candidate among the plurality of mouth shape candidates,
generate at least one second image based on the selected mouth shape candidate and at least a portion of each of the at least one first image, and
generate at least one third image by applying at least one super-resolution model to the at least one second image,
wherein the at least one processor is further configured to divide each of the at least one second image into at least one first area corresponding to a mouth and at least one second area including a remaining area except for the first area, generate at least one first high-resolution area by applying the at least one first area to a first super-resolution model specialized for the at least one first area, generate at least one second high-resolution area by applying the at least one second area to a second super-resolution model specialized for the at least one second area, and generate the at least one third image by synthesizing the at least one first high-resolution area with the at least one second high-resolution area, respectively.
16. A method of operating an electronic device, the method comprising:
acquiring voice data to be synthesized with at least one first image,
generating a plurality of mouth shape candidates by using the voice data,
selecting a mouth shape candidate among the plurality of mouth shape candidates,
generating at least one second image based on the selected mouth shape candidate and at least a portion of each of the at least one first image, and
generating at least one third image by applying at least one super-resolution model to the at least one second image,
wherein the generating of the at least one third image comprises:
dividing each of the at least one second image into at least one first area corresponding to a mouth and at least one second area including a remaining area except for the first area,
generating at least one first high-resolution area by applying the at least one first area to a first super-resolution model specialized for the at least one first area,
generating at least one second high-resolution area by applying the at least one second area to a second super-resolution model specialized for the at least one second area, and
generating the at least one third image by synthesizing the at least one first high-resolution area with the at least one second high-resolution area, respectively.
17. A non-transitory computer-readable storage medium storing at least one instruction, the at least one instruction causing at least one processor to:
acquire voice data to be synthesized with at least one first image,
generate a plurality of mouth shape candidates by using the voice data,
select a mouth shape candidate among the plurality of mouth shape candidates,
generate at least one second image based on the selected mouth shape candidate and at least a portion of each of the at least one first image, and
generate at least one third image by applying at least one super-resolution model to the at least one second image,
wherein the at least one instruction further causes the at least one processor to:
divide each of the at least one second image into at least one first area corresponding to a mouth and at least one second area including a remaining area except for the first area,
generate at least one first high-resolution area by applying the at least one first area to a first super-resolution model specialized for the at least one first area,
generate at least one second high-resolution area by applying the at least one second area to a second super-resolution model specialized for the at least one second area, and
generate the at least one third image by synthesizing the at least one first high-resolution area with the at least one second high-resolution area, respectively.