Intelligent mixing and replacing of persons in group portraits
The present disclosure is directed toward intelligently mixing and matching faces and/or people to generate an enhanced image that reduces or minimize artifacts and other defects. For example, the disclosed systems can selectively apply different alignment models to determine a relative alignment between a references image and a target image having an improved instance of the person. Upon aligning the digital images, the disclosed systems can intelligently identify a replacement region based on a boundary that includes the target instance and the reference instance of the person without intersecting other objects or people in the image. Using the size and shape of the replacement region around the target instance and the reference instance, the systems replace the instance of the person in the reference image with the target instance. The alignment of the images and the intelligent selection of the replacement region minimizes inconsistencies and/or artifacts in the final image.
1. A non-transitory computer readable medium storing instructions thereon that, when executed by at least one processor, cause a computing device to:
receive a user-defined plausible boundary about a reference instance of a person in a reference image from a plurality of images, wherein the plurality of images comprises a plurality of instances of the person and a target image with a target instance of the person,
align the target image and the reference image;
determine a replacement region by modifying the user-defined plausible boundary to generate a boundary that passes through consistent pixels in the target image and the reference image while passing as close as possible to the user-defined plausible boundary; and
generate an enhanced image by replacing a reference region from the reference image that corresponds with the replacement region with a target region from the target image that corresponds with the replacement region.
2. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to determine the replacement region using a cut-graph algorithm that identifies the boundary from the user-defined plausible boundary by identifying pixels with a maximum consistency and minimal passage across structured objects in the target and reference images.
3. The non-transitory computer readable medium of claim 2 , further comprising instructions that, when executed by the at least one processor, cause the computing device to constrain the cut-graph algorithm to pass as close as possible to the user-defined plausible boundary.
4. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
generate a plurality of aesthetic ratings for the plurality of instances of the person from the plurality of images; and
select an instance of the person with a highest aesthetic rating as the target instance.
5. The non-transitory computer readable medium of claim 1 , wherein the instructions, when executed by the at least one processor, cause the computing device to receive the user-defined plausible boundary about the reference instance of the person by receiving one or more touch gestures on a touch screen to select at least a portion of the reference instance of the person.
6. The non-transitory computer readable medium of claim 1 , wherein:
the user-defined plausible boundary defines a region surrounding a head of the person in the target image; and
replacing the reference region from the reference image with the target region from the target image comprises replacing the head of the person in the reference image with the head of the person from the target image.
7. The non-transitory computer readable medium of claim 1 , wherein:
the user-defined plausible boundary defines a region surrounding a head and torso of the person in the target image; and
replacing the reference region from the reference image with the target region from the target image comprises replacing the head of the person in the reference image with the head and torso of the person from the target image.
8. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to align the target image and the reference image by:
identifying a first gyroscope data set corresponding to the reference image and a second gyroscope data set corresponding to the target image;
detecting a first set of feature points within the reference image and a second set of feature points within the target image; and
applying an image-alignment model from a pixel-adjusted-gyroscope-alignment model and a feature-based-alignment model to align the target image with the reference image based on the first and second gyroscope data sets and one or both of the first and second sets of feature points.
9. A system operable to mix and match faces in group digital portraits, the system
comprising:
a camera;
a touch screen;
a plurality of images captured using the camera, the plurality of images including a plurality of instances of a person; and
at least one processor configured to cause the system to:
receive, via the touch screen, a user-defined plausible boundary about a reference instance of the person in a reference image from the plurality of images;
align a target image and the reference image;
determine a replacement region by modifying the user-defined plausible boundary to generate a boundary that passes through consistent pixels in the target image and the reference image while passing as close as possible to the user-defined plausible boundary; and
generate an enhanced image by replacing a reference region from the reference image that corresponds with the replacement region with a target region from the target image that corresponds with the replacement region.
10. The system of claim 9 , wherein the at least one processor is further configured to cause the system to capture the plurality of images using a burst mode of the camera.
11. The system of claim 9 , wherein the at least one processor is further configured to cause the system to determine the replacement region using a graph cut algorithm that identifies the boundary from the user-defined plausible boundary by identifying pixels with a maximum consistency and minimal passage across structured objects in the target and reference images while being constrained to have minimum deviation from the user-defined plausible boundary.
12. The system of claim 11 , wherein using the graph cut algorithm comprises penalizing potential boundaries going through high frequency regions more than potential boundaries going through low frequency areas.
13. The system of claim 9 , wherein the at least one processor is further configured to cause the system to align the target image and the reference image by:
selecting an image-alignment model from a pixel-adjusted-gyroscope-alignment model and a feature-based-alignment model; and
applying the selected image-alignment model to the target image and the reference image to determine an alignment between the target image and the reference image.
14. The system of claim 9 , wherein the at least one processor is further configured to cause the system to generate enhanced image by replacing the reference region with pixels form the target region using Gaussian blending.
15. The system of claim 14 , wherein using Gaussian blending comprises applying alpha blending on the target region and the reference region.
16. A method of mixing and matching faces in group digital portraits comprising:
receiving a user-defined plausible boundary about a reference instance of a person in a reference image from a plurality of images, wherein the plurality of images comprises a plurality of instances of the person and a target image with a target instance of the person;
aligning the target image and the reference image;
determining a replacement region by modifying the user-defined plausible boundary to generate a boundary that passes through consistent pixels in the target image and the reference image while minimizing deviation from the user-defined plausible boundary; and
generating an enhanced image by replacing a reference region from the reference image that corresponds with the replacement region with a target region from the target image that corresponds with the replacement region.
17. The method of claim 16 , further comprising receiving the user-defined plausible boundary about the reference instance of the person by receiving one or more touch gestures on a touch screen to select at least a portion of the reference instance of the person.
18. The method of claim 16 , wherein:
the user-defined plausible boundary defines a region surrounding a head of the person in the target image; and
replacing the reference region from the reference image with the target region from the target image comprises replacing the head of the person in the reference image with the head of the person from the target image.
19. The method of claim 16 , wherein:
the user-defined plausible boundary defines a region surrounding a head and torso of the person in the target image; and
replacing the reference region from the reference image with the target region from the target image comprises replacing the head of the person in the reference image with the head and torso of the person from the target image.
20. The method of claim 19 , further comprising:
receiving a second user-defined plausible boundary about a reference instance of a second person in the reference image;
determining a second replacement region based on the second user-defined plausible boundary, which includes the reference instance of the second person and a second target instance of the second person, having a second boundary passing through consistent pixels in the target image and the reference image; and
generating an updated enhanced image by replacing a second reference region from the reference image that corresponds with the second replacement region with a second target region from the target image that corresponds with the second replacement region.