Granted Patent B2

US 12,406,444 · App. 18/135,599 · Granted Sep 2, 2025

Inventors: Kai Zhou (Wiener Neudorf, AT); Laura Rosalia Luidolt (Vienna, AT); Himmy Tam (London, GB); Riza Alp Guler (London, GB); Iason Kokkinos (London, GB); Avihay Assouline (Tel Aviv, IL)

Assignee: Snap Inc.

G06T19/006G06T13/40G06T2210/16

Abstract

Claims (74)

1. A method comprising:

receiving, by one or more processors of a device, an image that includes a depiction of a real-world object having a fashion item in a real-world environment;

accessing a three-dimensional (3D) avatar model of a human;

generating a graphic item resembling the fashion item being worn by the real-world object depicted in the image;

modifying the 3D avatar model of the human based on the graphic item to generate a modified 3D avatar model, the modifying of the 3D avatar model comprising:

overlaying the graphic item on the 3D avatar model;

identifying an individual portion of the 3D avatar model that is not overlaid by the graphic item; and

reducing visibility of the individual portion of the 3D avatar model that is not overlaid by the graphic item; and

presenting the modified 3D avatar model within a view of the real-world environment on the device.

2. The method of claim 1 , wherein the graphic item comprises an augmented reality item, wherein the image is received from one or more cameras embedded in the device, and wherein the image comprises a frame of a video captured by the one or more cameras.

3. The method of claim 2 , wherein the modified 3D avatar model is added to the real-world environment depicted in the video, wherein reducing the visibility of the individual portion comprises making the individual portion at least partially transparent.

4. The method of claim 3 , wherein the modified 3D avatar model is presented within one or more lenses of AR glasses.

5. The method of claim 1 , wherein the real-world object comprises a person in the real-world environment.

6. The method of claim 1 , wherein the real-world object comprises a mannequin in the real-world environment.

7. The method of claim 1 , wherein the fashion item comprises an outfit, further comprising:

accessing an online inventory of a store that is within a threshold distance of the device that was used to capture the image that includes the depiction of the real-world object having the fashion item;

matching pixels of fashion item in the image with pixels of items in the inventory of the store;

identifying an individual item in the inventory of the store that matches the fashion item in the image; and

retrieving, as the graphic item, a detailed version of the fashion item using the identified individual item in the inventory of the store.

8. The method of claim 1 , further comprising:

receiving a user request to transfer the fashion item depicted in the image to the 3D avatar model, wherein the 3D avatar model is modified based on the graphic item to generate the modified 3D avatar in response to receiving the user request.

9. The method of claim 8 , wherein the user request comprises verbal input, a selection of an on-screen option, or a gesture detected in a video stream captured by the device.

10. The method of claim 1 , further comprising:

receiving input that selects the 3D avatar model from a plurality of 3D avatar models.

11. The method of claim 1 , further comprising:

segmenting the fashion item worn by the real-world object depicted in the image; and

applying a 3D cloth simulation model to the segmented fashion item to generate the graphic item.

12. The method of claim 1 , further comprising:

animating the 3D avatar model that has been modified based on the graphic item within the view of the real-world environment.

13. The method of claim 1 , further comprising:

loading the 3D avatar model in response to scanning a bar code that appears in the real-world environment; and

replacing one or more base garments worn by the 3D avatar model with the graphic item.

14. The method of claim 13 , further comprising:

overlaying the graphic item on the one or more base garments worn by the 3D avatar model; and

hiding portions of the one or more base garments that remain visible after being overlaid by the graphic item.

15. The method of claim 1 , further comprising:

performing a body scan to generate the 3D avatar model; and

receiving input that customizes a look of the generated 3D avatar model.

16. The method of claim 1 , further comprising:

presenting multiple copies of the 3D avatar model each being depicted as wearing a different fashion item, one of the copies of the 3D avatar model wearing the graphic item.

17. The method of claim 1 , further comprising:

determining a pose of the 3D avatar model;

obtaining a first set of body landmarks corresponding the 3D avatar model in the pose and a second set of body landmarks corresponding the real-world object wearing the fashion item;

computing a deviation between the first set of body landmarks and the second set of body landmarks;

modifying the first set of body landmarks associated with the real-world object to match the second set of body landmarks associated with the 3D avatar model based on the deviation;

applying a fitting model to the first and second sets of body landmarks to adjust one or more visual parameters of the graphic item corresponding to the modified first set of body landmarks;

generating an intermediate image by the fitting model depicting the graphic item with the adjusted one or more visual parameters overlaid on the 3D avatar model; and

applying a generative machine learning model to the intermediate image to blend sets of pixels corresponding to one or more gaps or occlusions that appear in the intermediate image.

18. The method of claim 17 , further comprising training the generative machine learning model by iterating through a sequence of training operations comprising:

receiving a first training image that depicts a training person in a first training pose and wearing a training fashion item;

receiving a training video that depicts the training person in a second training pose;

applying the generative machine learning model to the first training image and a given frame of the training video to generate a depiction of the training person in the second training pose wearing the training fashion item;

computing a deviation between the generated depiction of the training person in the second training pose wearing the training fashion item and the given frame of the training video; and

updating one or more parameters of the generative machine learning model based on the computed deviation.

19. A system comprising:

at least one processor of a device; and

a memory component having instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:

receiving an image that includes a depiction of a real-world object having a fashion item in a real-world environment;

accessing a three-dimensional (3D) avatar model of a human;

generating a graphic item resembling the fashion item being worn by the real-world object depicted in the image;

modifying the 3D avatar model of the human based on the graphic item to generate a modified 3D avatar model, the modifying of the 3D avatar model comprising:

overlaying the graphic item on the 3D avatar model;

identifying an individual portion of the 3D avatar model that is not overlaid by the graphic item; and

reducing visibility of the individual portion of the 3D avatar model that is not overlaid by the graphic item; and

presenting the modified 3D avatar model within a view of the real-world environment on the device.

20. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:

receiving, by a device, an image that includes a depiction of a real-world object having a fashion item in a real-world environment;

accessing a three-dimensional (3D) avatar model of a human;

generating a graphic item resembling the fashion item being worn by the real-world object depicted in the image;

modifying the 3D avatar model of the human based on the graphic item to generate a modified 3D avatar model, the modifying of the 3D avatar model comprising:

overlaying the graphic item on the 3D avatar model;

identifying an individual portion of the 3D avatar model that is not overlaid by the graphic item; and

reducing visibility of the individual portion of the 3D avatar model that is not overlaid by the graphic item; and

presenting the modified 3D avatar model within a view of the real-world environment on the device.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (31)

Cited By (1)

US 12,664,713