Computerized system and method for image creation using generative adversarial networks
Disclosed frameworks for generating an image including a salient object and a staged background include extracting a salient object from a source image and applying a generative model to the salient object to generate the image. According to some embodiments, extracting a salient object from a source image involves using salient object detection method to identify the relevant portions of the source image corresponding to the salient object. In some embodiments, the generative model is a generative adversarial network trained using a domain relevant dataset.
1. A method comprising:
retrieving a first image corresponding to a domain, the first image including a first salient object;
retrieving a second image corresponding to the domain, the second image including a second salient object;
generating a segmented background by removing the second salient object from the second image, the segmented background having removed pixels;
generating a target background by inpainting the removed pixels of the segmented background; and
generating a third image including the first salient object and the target background by aligning a shape and a center mass of a mask of the first salient object to a mask of the second salient object.
2. The method of claim 1 , wherein the step of generating a segmented background comprises detecting and segmenting the second salient object using a salient object detection model.
3. The method of claim 1 , further comprising extracting, by a salient object detection model, the first salient object from the first image.
4. The method of claim 1 , wherein the step of generating a target background by inpainting is performed by an inpainting process using at least one neural network having a loss function comprising a weighted boundary loss.
5. The method of claim 1 , further comprising animating the third image, the step of animating comprising:
generating a salient object animation frame by:
segmenting the first salient object from the third image;
translating the first salient object from a first position to a second position; and,
inpainting blank pixels in the salient object animation frame using an inpainting process;
generating an N number of salient object animation frames by repeating the steps of segmenting, translating, and inpainting; and,
generating a salient object animation comprising at least some of the salient object animation frames.
6. The method of claim 5 , wherein the inpainting process uses at least one neural network having a loss function comprising a weighted boundary loss.
7. A method comprising:
retrieving a first image corresponding to a domain, the first image including a first salient object;
retrieving a domain relevant dataset corresponding to the domain, the domain relevant dataset including a plurality of second images, the second images having second salient objects;
ranking the plurality of second images based on a similarity measure of each second image to the first image;
selecting a subset of second images from the plurality of second images based on the ranking;
generating a set of segmented backgrounds by removing the second salient objects from each second image from the subset of second images, each of the segmented background of the set of segmented backgrounds having removed pixels;
generating a set of target backgrounds by inpainting the removed pixels of each segmented background of the set of segmented backgrounds; and,
generating a set of third images each image of the set of third images including the first salient object and a target background from the set of target backgrounds by aligning a shape and a center mass of a mask of the first salient object to a mask of each of the second salient objects of the second images of the subset of second images.
8. The method of claim 7 , wherein the step of generating a set of segmented backgrounds comprises detecting and segmenting the second salient objects using a salient object detection model.
9. The method of claim 7 , further comprising extracting, by a salient object detection model, the first salient object from the first image.
10. The method of claim 7 , wherein the step of generating a set of target backgrounds by inpainting is performed by an inpainting process using at least one neural network having a loss function comprising a weighted boundary loss.
11. The method of claim 7 , further comprising animating each of the set of third images, the step of animating comprising:
generating a salient object animation frame by:
segmenting the first salient object from a third image from the set of third images;
translating the first salient object from a first position to a second position; and,
inpainting blank pixels in the salient object animation frame using an inpainting process;
generating an N number of salient object animation frames by repeating the steps of segmenting, translating, and inpainting; and,
generating a salient object animation comprising at least some of the salient object animation frames.
12. The method of claim 11 , wherein the inpainting process uses at least one neural network having a loss function comprising a weighted boundary loss.
13. A non-transitory computer-readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining steps of:
retrieving a first image corresponding to a domain, the first image including a first salient object;
retrieving a second image corresponding to the domain, the second image including a second salient object;
generating a segmented background by removing the second salient object from the second image, the segmented background having removed pixels;
generating a target background by inpainting the removed pixels of the segmented background; and
generating a third image including the first salient object and the target background by aligning a shape and a center mass of a mask of the first salient object to a mask of the second salient object.
14. The non-transitory computer-readable storage medium of claim 13 , wherein the step of generating a segmented background comprises detecting and segmenting the second salient object using a salient object detection model.
15. The non-transitory computer-readable storage medium of claim 13 , further comprising extracting, by a salient object detection model, the first salient object from the first image.
16. The non-transitory computer-readable storage medium of claim 13 , wherein the step of generating a target background by inpainting is performed by an inpainting process using at least one neural network having a loss function comprising a weighted boundary loss.
17. The non-transitory computer-readable storage medium of claim 13 , further comprising:
retrieving a domain relevant dataset corresponding to the domain, the domain relevant dataset including a plurality of second images, the second images having second salient objects;
ranking the plurality of second images based on a similarity measure of each second image to the first image;
selecting a subset of second images from the plurality of second images based on the ranking;
generating a set of segmented backgrounds by removing the second salient objects from each second image from the subset of second images, each of the segmented background of the set of segmented backgrounds having removed pixels;
generating a set of target backgrounds by inpainting the removed pixels of each segmented background of the set of segmented backgrounds; and,
generating a set of fourth images each image of the set of fourth images including the first salient object and a target background from the set of target backgrounds.