Data-driven, photorealistic social face-trait encoding, prediction, and manipulation using deep neural networks
When one looks at a face, one cannot help but ‘read’ it: in the blink of an eye, people form reliable impressions of both transient psychological states (e.g., happiness) and stable character traits (e.g., trustworthiness). Such impressions are irresistible, formed with high levels of consensus, and important for social decisions. Disclosed herein is a large-scale data-driven methodology that allows for the easy manipulation of social trait information in hyper-realistic face images. For example, a given face image could be made to look more or less trustworthy by moving a simple slider. Further, this method can not only generate faces, but can ‘read’ faces as well, providing confidence estimates of different social traits for any arbitrary image. The disclosed approach is both fast and accurate, and represents a paradigm shift in facial photo manipulation.
1. A system for photorealistic social face-trait encoding, prediction, and manipulation, comprising one or more processors configured with machine-readable instructions, that when executed cause the one or more processors to:
a. encode an image of an original face as a multi-dimensional vector comprising one or more learned image features using a two-stage encoding process;
b. modify the multi-dimensional vector to adjust at least one subjective social trait based on a learned function between the subjective social trait and the one or more learned image features in the multi-dimensional vector; and
c. decode the modified multi-dimensional vector to generate an image of a realistic synthetic face.
2. The system according to claim 1 , wherein the one or more processors are further configured with machine-readable instructions, that when executed cause the one or more processors to establish the correlation between the subjective social trait and the one or more learned image features the adjustments to the multi-dimensional vector are based upon.
3. The system according to claim 2 , wherein the correlation is established based on a dataset of a plurality of images, each of which is rated by a plurality of subjects for the subjective social trait.
4. The system according to claim 3 , wherein the plurality of images comprises at least 1,000 images, and the plurality of subjects comprises at least 25 subjects.
5. The system according to claim 1 , wherein the correlation between the subjective social trait and the one or more learned image features is established based on at least one other correlation between a different subjective social trait and the one or more learned image feature.
6. The system according to claim 1 , wherein the one or more processors are further configured with machine-readable instructions, that when executed cause the one or more processors to estimate at least one feature of the original face.
7. The system according to claim 6 , wherein the at least one estimated feature is selected from the group of features consisting of age, gender, level of perceived trustworthiness, and level of perceived smiling.
8. The system according to claim 1 , wherein the one or more processors are present on one or more servers.
9. The system according to claim 1 , wherein each of the one or more processors are a graphics processing unit (GPU).
10. The system according to claim 1 , wherein the one or more processors are further configured with machine-readable instructions, that when executed cause the one or more processors to transmit the realistic synthetic face to a mobile device.
11. The system according to claim 1 , wherein the one or more processors are further configured with machine-readable instructions, that when executed cause the one or more processors to receive the image of the original face and a one or more instructions, and wherein modifying the multi-dimensional vector is further based on the one or more instructions.
12. The system according to claim 1 , wherein the one or more processors are further configured with machine-readable instructions, that when executed cause the one or more processors to modify the multi-dimensional vector of the encoded original face a plurality of times, each time generating a different image of a realistic synthetic faces, each based on a different adjustment of the at least one subjective social trait.
13. The system according to claim 1 , wherein two-stage encoding process comprises a fast encoding component that is trained to predict image features given an accompanying image, and a slow encoding component having a gradient descent optimization algorithm that uses the predicted image features as the initializations, where the slow encoding component searches for image features that would result in the realistic synthetic face to have similar features as the original face.
14. The system according to claim 1 , wherein decoding is performed by a neural network trained to generate realistic synthetic faces based on the multi-dimensional vector of learned image feature.
15. The system according to claim 1 , wherein the at least one subjective social trait comprises trustworthiness or dominance.
16. The system according to claim 1 , wherein the one or more processors are further configured to modify the multi-dimensional vector to adjust an objective appearance-based dimension.
17. The system according to claim 1 , wherein the one or more processors are further configured to map an arbitrary semantic trait representation to a corresponding visual trait representation.
18. A method for photorealistic social face-trait encoding, prediction, and manipulation, comprising the steps of:
a. receiving an image of a face;
b. mapping the image of the face to a multi-dimensional vector of learned image features using a first neural network trained to map any image of any face to a multi-dimensional vector of learned image feature;
c. adjusting a subjective social trait dimension of the image of the face by adjusting a value of at least one learned image feature based on a linear correlation established between the social trait dimension and the at least one learned image feature;
d. generating a realistic synthetic face based on the mapped image of the face and the adjusted values of the at least one learned image feature using a second neural network trained to generate realistic synthetic faces based on the multi-dimensional vector of learned image feature.