Granted Patent B2

US 12,586,272 · App. 18/332,869 · Granted Mar 24, 2026

Inventors: Xin Chung (Bolingbrook, IL); Bruce V. Schwartz (Bolingbrook, IL); Giorgio Patrini (Bolingbrook, IL); Ajinkya Indulkar (Bolingbrook, IL); Ghassen Chaabouni (Bolingbrook, IL); Alberto Gaona (Bolingbrook, IL); Juan Cardelino (Montevideo, UY)

Assignee: Ulta Salon, Cosmetics & Fragrance, Inc.

G06T11/60G06T5/50G06T5/77G06T7/194G06T11/001G06T2207/20021G06T2207/20084G06T2207/20212G06T2207/30201G06T2210/44

Abstract

Claims (82)

1 . A computer-implemented image processing method, comprising:

receiving an original digital image including a face area of a first subject and a background area;

receiving a reference image of a second subject;

modifying the original digital image based on the reference image utilizing a generative neural network to create a modified original digital image in which a feature of the second subject is transferred to the first subject;

extracting a feature area from the modified original digital image including a portion of the modified original digital image with the feature of the second subject transferred to the first subject, and extracting the face area of the first subject and the background area from the original digital image; and

replacing a portion of modified face and background areas of the modified original digital image with the face area of the first subject and the background area from the original digital image to create a composite modified digital image that includes (i) data representing the feature of the second subject transferred to the first subject as extracted from the modified original digital image and (ii) data representing the face area of the first subject and the background area as extracted from the original digital image.

2 . The method of claim 1 , wherein modifying the original digital image based on the reference image utilizing the generative neural network to create the modified original digital image comprises:

piecewise morphing of the reference image such that dimensions of components of the second subject are morphed into dimensions of components of the first subject to create a morphed reference image.

3 . The method of claim 2 , further comprising:

receiving a plurality of reference images of a plurality of subjects, wherein the reference image of the second subject is one of the plurality of reference images;

piecewise morphing of the plurality of reference images such that dimensions of components of the plurality of subjects are morphed into dimensions of components of the first subject to create a plurality of morphed reference images, wherein the morphed reference image is one of the plurality of morphed reference images; and

based on selection of one of the plurality of reference images, outputting a corresponding one of the plurality of morphed reference images as the modified original digital image.

4 . The method of claim 2 , further comprising:

superimposing the face area of the first subject and the background with the morphed reference image to create the modified original digital image.

5 . The method of claim 1 , wherein modifying the original digital image based on the reference image utilizing the generative neural network comprises:

projecting the original digital image into a latent vector space to create a vector image;

modifying the vector image based on the reference image utilizing the generative neural network to create a modified vector image; and

inverting the modified vector image into the modified original digital image.

6 . The method of claim 1 , wherein modifying the original digital image based on the reference image utilizing the generative neural network to create the modified digital image comprises:

modifying the original digital image to transfer hair of the second subject to the first subject.

7 . The method of claim 1 , wherein:

receiving the original digital image including the face area of the first subject comprises receiving the original digital image including the face area of a first real-world person;

receiving the reference image of the second subject comprises receiving the reference image of a second real-world person; and

modifying the original digital image based on the reference image utilizing the generative neural network to create the modified original digital image comprises transferring the feature of the second real-world person to the first real-world person.

8 . The method of claim 1 , wherein:

receiving the original digital image including the face area of the first subject comprises receiving the original digital image including the face area of a first real-world person;

receiving the reference image of the second subject comprises receiving the reference image of a second virtual person or avatar; and

modifying the original digital image based on the reference image utilizing the generative neural network to create the modified original digital image comprises transferring the feature of the second virtual person or avatar to the first real-world person.

9 . The method of claim 1 , wherein:

receiving the original digital image including the face area of the first subject comprises receiving the original digital image including the face area of a first virtual person or avatar;

receiving the reference image of the second subject comprises receiving the reference image of a second real-world person; and

modifying the original digital image based on the reference image utilizing the generative neural network to create the modified original digital image comprises transferring the feature of the second real-world person to the first virtual person or avatar.

10 . The method of claim 1 , wherein:

receiving the original digital image including the face area of the first subject comprises receiving the original digital image including the face area of a virtual person or avatar;

receiving the reference image of the second subject comprises receiving the reference image of a second virtual person or avatar; and

modifying the original digital image based on the reference image utilizing the generative neural network to create the modified original digital image comprises transferring the feature of the second virtual person or avatar to the first virtual person or avatar.

11 . The method of claim 1 , wherein extracting the feature area from the modified original digital image, and the face area of the first subject and the background area from the original digital image comprises:

performing mask segmentation of (i) the original digital image to segment the face area of the first subject and the background area and of (ii) the reference image to segment the feature of the second subject.

12 . The method of claim 1 , further comprising:

inpainting the composite modified digital image to preserve a photo-realistic view of the first subject.

13 . The method of claim 12 , wherein inpainting the composite modified digital image comprises:

based on a comparison of the original digital image, the modified original digital image, and the composite modified digital image, filling in an area of the feature area within the modified original digital image; and

preserving an identity of the first subject for a photo-realistic view of the first subject in the composite modified digital image.

14 . The method of claim 12 , wherein inpainting the composite modified digital image comprises:

performing segmentation of the composite modified digital image to segment the background area and the feature area into a reference image;

digitally filling in an area of the feature area within the reference image; and

positioning the face area of the first subject from the digital image into the reference image for a pasted reference image.

15 . The method of claim 14 , further comprising:

using a face segmenting mask, positioning a forehead and hairline area from the composite modified digital image into the pasted reference image to create a pre-output image.

16 . The method of claim 15 , further comprising:

modifying the pre-output image to include clothing based on a length of hair in the digital image in relation to a shoulder area of the first subject.

17 . The method of claim 1 , further comprising:

modifying a color balance of the composite modified digital image such that components from the original digital image in the composite modified digital image substantially match a perceived quality of components from the reference image in the composite modified digital image.

18 . The method of claim 1 , further comprising:

modifying a color balance of the composite modified digital image such that components from the reference image in the composite modified digital image substantially match a perceived quality of components from the original digital image in the composite modified digital image.

19 . The method of claim 1 , further comprising:

modifying a color balance of the composite modified digital image to maintain a perceived quality of the original digital image in the composite modified digital image.

20 . The method of claim 1 , further comprising:

determining first scene conditions of the original digital image, wherein the scene conditions include lighting features across the face area of the first subject; and

modifying second scene condition of the composite modified digital image to substantially match the first scene conditions of the original digital image.

21 . The method of claim 1 , further comprising:

performing pre-processing of the original digital image for at least one of aligning, cropping, and resizing the digital image prior to modifying the original digital image utilizing the generative neural network.

22 . A non-transitory computer-readable media having stored therein executable instructions, which when executed by a system including one or more processors causes the system to perform functions comprising:

receiving an original digital image including a face area of a first subject and a background area;

receiving a reference image of a second subject;

modifying the original digital image based on the reference image utilizing a generative neural network to create a modified original digital image in which a feature of the second subject is transferred to the first subject;

extracting a feature area from the modified original digital image including a portion of the modified original digital image with the feature of the second subject transferred to the first subject, and extracting the face area of the first subject and the background area from the original digital image; and

replacing a portion of modified face and background areas of the modified original digital image with the face area of the first subject and the background area from the original digital image to create a composite modified digital image that includes (i) data representing the feature of the second subject transferred to the first subject as extracted from the modified original digital image and (ii) data representing the face area of the first subject and the background area as extracted from the original digital image.

23 . The non-transitory computer-readable media of claim 22 , wherein the functions further comprise:

receiving a plurality of reference images of a plurality of subjects, wherein the reference image of the second subject is one of the plurality of reference images; and

wherein modifying the original digital image based on the reference image utilizing the generative neural network to create the modified original digital image comprises:

piecewise morphing of the plurality of reference images such that dimensions of components of the plurality of subjects are morphed into dimensions of components of the first subject to create a plurality of morphed reference images; and

based on selection of one of the plurality of reference images, outputting a corresponding one of the plurality of morphed reference images as the modified digital image.

24 . A system comprising:

one or more processors and non-transitory computer-readable media having stored therein instructions, which when executed by the one or more processors, causes the system to perform functions comprising:

receiving an original digital image including a face area of a first subject and a background area;

receiving a reference image of a second subject;

modifying the original digital image based on the reference image utilizing a generative neural network to create a modified original digital image in which a feature of the second subject is transferred to the first subject;

extracting a feature area from the modified original digital image including a portion of the modified original digital image with the feature of the second subject transferred to the first subject, and extracting the face area of the first subject and the background area from the original digital image; and

replacing a portion of modified face and background areas of the modified original digital image with the face area of the first subject and the background area from the original digital image to create a composite modified digital image that includes (i) data representing the feature of the second subject transferred to the first subject as extracted from the modified original digital image and (ii) data representing the face area of the first subject and the background area as extracted from the original digital image.

25 . The system of claim 24 , wherein the functions further comprise:

modifying a color balance of the composite modified digital image such that components from the original digital image in the composite modified digital image substantially match a perceived quality of components from the reference image in the composite modified digital image.

Assignments (1)

Continuity (1)

References Cited (21)