Image realism predictor
A method for predicting the realism of an object within an image includes generating a training image set for a predetermined object type. The training image set comprises one or more training images at least partially generated using a computer. A pixel level training spatial realism map is generated for each training image of the one or more training images. Each training spatial realism map configured to represent a perceptual realism of the corresponding training image. A predictor is trained using the training image set and the corresponding training spatial realism maps. An image of the predetermined object is received. A spatial realism map of the received image is produced using the trained predictor.
1. A method for predicting the realism of an object within an image, comprising:
generating a training image set for a predetermined object type, the training image set comprising one or more training images at least partially generated using a computer;
generating a pixel level training spatial realism map for each training image of the one or more training images, each training spatial realism map configured to represent a perceptual realism of a corresponding training image;
training a predictor using the training image set and corresponding training spatial realism maps;
receiving an image of the predetermined object type; and
producing a spatial realism map of the received image using the trained predictor.
2. The method of claim 1 , wherein generating a training image set for the predetermined object type comprises distorting at least a portion of a natural image, and wherein a corresponding spatial realism map is defined to have a low realism score in the distorted portion of the natural image and high realism score in undistorted portions.
3. The method of claim 2 , wherein distorting at least a portion of the natural image comprises swapping at least the portion of the natural image with a corresponding portion of a computer-generated image to create a combined image.
4. The method of claim 3 , further comprising smoothly blending the natural image and the computer-generated image.
5. The method of claim 1 , wherein generating a training image set and spatial realism map set for the predetermined object type comprises:
presenting, to a user, each training image of the training image set;
receiving an annotation of each training image from the user; and
generating a pixel level spatial realism map based on the received annotation.
6. The method of claim 5 wherein the annotation comprises one or more marked regions in each training image that appear unrealistic to the user.
7. The method of claim 6 wherein the annotation comprises one or more of a bounding polygon, a circle, and an ellipse.
8. The method of claim 1 , wherein generating a training image set comprises generating the training image set using one or more of a deep convolutional generative adversarial network, a self-attention generative adversarial network (SAGAN), and a boundary equilibrium generative adversarial network (BEGAN).
9. The method of claim 1 , wherein the predetermined object type is a human face.
10. The method of claim 1 , wherein the predictor is implemented as a deep convolutional neural network.
11. The method of claim 1 , wherein the predictor is implemented as a U-Net deep neural network.
12. An image realism predictor, comprising:
a processor; and
a memory storing computer program instructions which when executed by the processor cause the processor to perform operations comprising:
generating a training image set for a predetermined object type, the training image set comprising one or more training images at least partially generated using a computer;
generating a pixel level training spatial realism map for each training image of the one or more training images, each training spatial realism map configured to map a perceptual realism of a corresponding training image;
training a predictor using the using the training image set and corresponding training spatial realism maps;
receiving an image of the predetermined object type; and
producing a spatial realism map of the received image using the trained predictor.
13. The image realism predictor of claim 12 , wherein generating a training image for the predetermined object type comprises distorting at least a portion of a natural image, and wherein a corresponding spatial realism map is defined to have a low realism score in the distorted portion of the natural image and high realism score in undistorted portions.
14. The image realism predictor of claim 13 , wherein distorting at least a portion of the natural image comprises swapping at least the portion of the natural image with a corresponding portion of a computer-generated image to create a combined image.
15. The image realism predictor of claim 14 , wherein the processor is configured to blend the natural image and the computer-generated image.
16. The image realism predictor of claim 12 , wherein generating a training image set and spatial realism map set for the predetermined object type comprises:
presenting, to a user, each training image of the training image set;
receiving an annotation of each training image from the user; and
generating a pixel level spatial realism map based on the received annotation.
17. The image realism predictor of claim 16 , wherein the annotation comprises one or more marked regions in each training image that appear unrealistic to the user.
18. The image realism predictor of claim 17 , wherein the annotation comprises one or more of a bounding polygon, a circle, and an ellipse.
19. The image realism predictor of claim 12 , wherein the processor is configured to generate the training image set using one or more of a deep convolutional generative adversarial network, a self-attention generative adversarial network (SAGAN), and a boundary equilibrium generative adversarial network (BEGAN).