Apparatus and method for generating speech synthesis image
An apparatus for generating a speech synthesis image according to a disclosed embodiment is an apparatus for generating a speech synthesis image based on machine learning, the apparatus including a first global geometric transformation predictor configured to be trained to receive each of a source image and a target image including the same person, and predict a global geometric transformation for a global motion of the person between the source image and the target image based on the source image and the target image, a local feature tensor predictor configured to be trained to predict a feature tensor for a local motion of the person based on input target image-related information, and an image generator configured to be trained to reconstruct the target image based on the global geometric transformation, the source image, and the feature tensor for the local motion.
1. An apparatus for generating a speech synthesis image based on machine learning, the apparatus comprising:
a first global geometric transformation predictor configured to be trained to receive each of a source image and a target image including the same person, and predict a global geometric transformation for a global motion of the person between the source image and the target image, based on the source image and the target image;
a local feature tensor predictor configured to be trained to predict a feature tensor for a local motion of the person, based on input target image-related information; and
an image generator configured to be trained to reconstruct the target image, based on the global geometric transformation, the source image, and the feature tensor for the local motion,
wherein the global motion is a motion of the person with an amount greater than or equal to a preset threshold amount of motion,
wherein the first global geometric transformation predictor is further configured to:
extract a source image heat map based on the source image, the source image heat map being a probability distribution map in an image space indicating whether each pixel in the source image is a pixel related to the global motion of the person;
extract a target image heat map based on the target image, the target image heat map being a probability distribution map in the image space as to whether each pixel in the target image is a pixel related to the global motion of the person; and
calculate the global geometric transformation based on the source image heat map and the target image heat map.
2. The apparatus according to claim 1 , wherein
the local motion is a motion of a face when the person is speaking.
3. The apparatus according to claim 2 , wherein the first global geometric transformation predictor is further configured to extract a geometric transformation into the source image heat map from a preset reference probability distribution, based on the source image, extract a geometric transformation into the target image heat map from the preset reference probability distribution, based on the target image, and calculate the global geometric transformation, based on the geometric transformation into the source image heat map from the reference probability distribution and the geometric transformation into the target image heat map from the reference probability distribution.
4. The apparatus according to claim 2 , wherein the local feature tensor predictor includes a first local feature tensor predictor configured to be trained to predict a speech feature tensor for a local speech motion of the person, based on the input target image-related information, and
the local speech motion is a motion related to a speech of the local motion of the person.
5. The apparatus according to claim 4 , wherein the local feature tensor predictor further includes a second local feature tensor predictor configured to be trained to predict a non-speech feature tensor for a local non-speech motion of the person, based on the input target image-related information, and
the local non-speech motion is a motion not related to the speech of the local motion of the person.
6. The apparatus according to claim 5 , wherein the second local feature tensor predictor is trained to receive a target partial image including only a motion not related to speech of the person in the target image, and predict the non-speech feature tensor, based on the target partial image.
7. The apparatus according to claim 2 , further comprising an optical flow predictor configured to be trained to calculate an optical flow between the source image and the target image, based on the source image and the global geometric transformation,
wherein the image generator is trained to reconstruct the target image, based on the optical flow between the source image and the target image, the source image, and the feature tensor for the local motion.
8. The apparatus according to claim 1 , wherein the first global geometric transformation predictor is further configured to calculate a geometric transformation into any i-th (iϵ{1, 2, . . . , n}) (n is a natural number equal to or greater than 2) frame heat map in an image having n frames from a preset reference probability distribution when the image is input, and calculate a global geometric transformation between two adjacent frames in the image, based on the geometric transformation into the i-th frame heat map from the reference probability distribution.
9. The apparatus according to claim 8 , further comprising a second global geometric transformation predictor configured to receive sequential voice signals corresponding to the n frames, and to be trained to predict a global geometric transformation between two adjacent frames in the image from the sequential voice signals.
10. The apparatus according to claim 9 , wherein the second global geometric transformation predictor is further configured to adjust a parameter of an artificial neural network to minimize a difference between the global geometric transformation between the two adjacent frames which is predicted in the second global geometric transformation predictor and the global geometric transformation between the two adjacent frames which is calculated in the first global geometric transformation predictor.
11. The apparatus according to claim 10 , wherein in a test process for speech synthesis image generation,
the second global geometric transformation predictor is further configured to receive sequential voice signals of a person, calculate a global geometric transformation between two adjacent frames in an image corresponding to the sequential voice signals from the sequential voice signals, and calculate a global geometric transformation between a preset target frame and a preset start frame, based on the global geometric transformation between the two adjacent frames,
the local feature tensor predictor is further configured to predict the feature tensor for the local motion of the person, based on the input target image-related information, and
the image generator is further configured to reconstruct the target frame, based on the global geometric transformation, the source image, and the feature tensor for the local motion.
12. A method for generating a speech synthesis image, based on machine learning that is performed in a computing device including one or more processors and a memory storing one or more programs executed by the one or more processors, the method comprising:
training a first global geometric transformation predictor to receive each of a source image and a target image including the same person, and predict a global geometric transformation for a global motion of the person between the source image and the target image, based on the source image and the target image;
training a local feature tensor predictor to predict a feature tensor for a local motion of the person, based on input target image-related information; and
training an image generator to reconstruct the target image, based on the global geometric transformation, the source image, and the feature tensor for the local motion,
wherein the global motion is a motion of the person with an amount greater than or equal to a preset threshold amount of motion,
wherein the training of the first global geometric transformation predictor comprises:
extracting a source image heat map based on the source image, the source image heat map being a probability distribution map in an image space indicating whether each pixel in the source image is a pixel related to the global motion of the person;
extracting a target image heat map based on the target image, the target image heat map being a probability distribution map in the image space as to whether each pixel in the target image is a pixel related to the global motion of the person; and
calculating the global geometric transformation based on the source image heat map and the target image heat map.