Optimization of overall editing vector to achieve target expression photo editing effect
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for automatically generating datasets for a particular target expression photo effect. In one aspect, a system comprises receiving a plurality of image pairs, each comprising an original face image and an expressive face image representative of a target expression photo editing effect, generating an initial overall editing vector, wherein generating the initial overall editing vector comprises processing each image pair using a style space encoder model to generate an embedding of the original face image and an embedding of the expressive face image in an embedding space, optimizing the initial overall editing vector in accordance with one or more optimization criteria to generate an optimized overall editing vector, and applying the optimized overall editing vector to an input face image to generate a target expression face image that has the target expression photo editing effect.
1 . A computer-implemented method comprising:
receiving a plurality of image pairs, wherein each image pair comprises an original face image of a face and an expressive face image representative of a target expression photo editing effect applied to the face;
generating an initial overall editing vector, wherein generating the initial overall editing vector comprises processing each image pair using a style space encoder model to generate an embedding of the original face image and an embedding of the expressive face image in an embedding space, wherein the embedding space comprises a plurality of style dimensions, and wherein each style dimension represents characteristics of a respective facial expression type;
optimizing the initial overall editing vector in accordance with one or more optimization criteria to generate an optimized overall editing vector; and
applying the optimized overall editing vector to an input face image to generate a target expression face image that has the target expression photo editing effect.
2 . The method of claim 1 , wherein, for each image pair, the original face image in the image pair has been edited to generate the expressive face image in the image pair.
3 . The method of claim 1 , wherein the style space encoder model is an encoder of a pretrained style transfer neural network.
4 . The method of claim 1 , wherein generating the initial overall editing vector further comprises determining a respective editing vector for each image pair characterizing a difference between the embedding of the original face image from the embedding of the expressive face image.
5 . The method of claim 4 , further comprising, for a respective plurality of editing vectors corresponding with the plurality of image pairs, calculating a mean of the plurality of editing vectors as the initial overall editing vector.
6 . The method of claim 1 , wherein generating the initial overall editing vector further comprises:
determining a normal vector separating a hyperplane between embeddings of original face images and embeddings of expressive face images for the plurality of image pairs as the initial overall editing vector.
7 . The method of claim 6 , wherein determining the normal vector separating the hyperplane further comprises using a support vector machine to perform binary classification.
8 . The method of claim 1 , wherein optimizing the initial overall editing vector in accordance with one or more optimization criteria to generate the optimized overall editing vector further comprises performing gradient optimization comprising:
processing the original face image for each image pair in the plurality of image pairs, wherein processing comprises:
using the style space encoder model to generate the embedding of the original face image in the embedding space;
applying an overall editing vector to the embedding of the original face image to generate a generated expression embedding;
decoding the generated expression embedding into a generated expression image using a decoder model;
computing a set of one or more losses based at least on a set of one or more generated expression images; and
updating the overall editing vector.
9 . The method of claim 8 , wherein computing the set of one or more losses based at least on a set of one or more generated expression images comprises:
computing a regularization loss with respect to each corresponding expressive face image for each processed original face image;
computing a Laplacian loss indicative of a measure of generated expression image sharpness; and
using a set of one or more loss models, each configured to generate a respective loss for the generated expression image with respect to a corresponding criterion.
10 . The method of claim 9 , wherein the set of one or more loss models comprises:
a first subset of loss models configured to process the generated expression image and the corresponding expressive face image to generate a first set of losses; and
a second subset of loss models configured to process the generated expression image to generate a second set of losses.
11 . The method of claim 10 , wherein the first subset of loss models comprises a perceptual similarity model configured to generate a learned perceptual image patch loss, wherein each patch comprises a region in each image, as a measure of matching content between the generated expression image and the corresponding expressive face image.
12 . The method of claim 10 , wherein the second subset of loss models comprises:
a neural image assessment model configured to generate a quality score indicative of at least a measure of artifacts in the generated expression image; and
a blend shape model configured to generate a probability for each of a plurality of facial phase transitions for each of a plurality of facial landmarks identified in the generated expression image.
13 . The method of claim 12 , wherein using the blend shape model further comprises defining a first facial phase transition by assigning one or more values to a subset of the plurality of facial landmarks identified in the generated expression image.
14 . The method of claim 10 , wherein the first and second set of losses are each multiplied with a corresponding first and second set of loss weight parameters before updating the overall editing vector.
15 . The method of claim 14 , further comprising tuning the first and second set of loss weight parameters using hyperparameter tuning.
16 . The method of claim 1 , further comprising postprocessing of the optimized overall editing vector using one or more of trimming, clipping, or normalization.
17 . The method of claim 1 , wherein applying the optimized editing vector to the input face image to generate a target expression face image in accordance with the target expression photo editing effect further comprises:
processing the input image using the style space encoder model to generate an embedding of the input face image in the embedding space;
applying the optimized overall editing vector to the embedding of the input face image to generate a target expression face embedding; and
decoding the target expression face embedding into the target expression face image using a decoder model.
18 . The method of claim 17 , further comprising including the target expression face image in a dataset of target expression face images.
19 . The method of claim 18 , further comprising training a student model on the dataset of target expression face images.
20 . The method of claim 19 , further comprising providing the trained student model for deployment on an at-edge device to generate the target expression face images on the at-edge device.
21 . The method of claim 20 , further comprising deploying the trained student model on the at-edge device.
22 . A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers to cause the one or more computers to perform operations comprising:
receiving a plurality of image pairs, wherein each image pair comprises an original face image of a face and an expressive face image representative of a target expression photo editing effect applied to the face;
generating an initial overall editing vector, wherein generating the initial overall editing vector comprises processing each image pair using a style space encoder model to generate an embedding of the original face image and an embedding of the expressive face image in an embedding space, wherein the embedding space comprises a plurality of style dimensions, and wherein each style dimension represents characteristics of a respective facial expression type;
optimizing the initial overall editing vector in accordance with one or more optimization criteria to generate an optimized overall editing vector; and
applying the optimized overall editing vector to an input face image to generate a target expression face image that has the target expression photo editing effect.
23 . The system of claim 22 , wherein generating the initial overall editing vector further comprises determining a respective editing vector for each image pair characterizing a difference between the embedding of the original face image from the embedding of the expressive face image.
24 . A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that are operable, when executed by data processing apparatus to cause the data processing apparatus to perform operations comprising:
receiving a plurality of image pairs, wherein each image pair comprises an original face image of a face and an expressive face image representative of a target expression photo editing effect applied to the face;
generating an initial overall editing vector, wherein generating the initial overall editing vector comprises processing each image pair using a style space encoder model to generate an embedding of the original face image and an embedding of the expressive face image in an embedding space, wherein the embedding space comprises a plurality of style dimensions, and wherein each style dimension represents characteristics of a respective facial expression type;
optimizing the initial overall editing vector in accordance with one or more optimization criteria to generate an optimized overall editing vector; and
applying the optimized overall editing vector to an input face image to generate a target expression face image that has the target expression photo editing effect.