Granted Patent B2

US 12,602,842 · App. 18/529,550 · Granted Apr 14, 2026

Inventors: Bohdan Ahafonov (Santa Monica, CA); Matthew Hallberg (Los Angeles, CA); Sergei Korolev (Marina del Rey, CA); William Miles Miller (San Francisco, CA); Daria Skrypnyk (Los Angeles, CA); Aleksei Stoliar (Marina del Rey, CA)

Assignee: Snap Inc.

G06T11/001G06T7/11G06T11/40G06T11/60G06V20/20G10L15/08G10L15/22G06T2210/16G10L2015/088

Abstract

Claims (60)

1 . A method comprising:

storing, in a multimodal memory, interaction data representing use of one or more interaction functions including data in different modalities;

detecting an object depicted in an image captured by an interaction client;

generating, by a machine learning model, a prompt based on the object depicted in the image and the interaction data in the multimodal memory;

generating an artificial texture based on the prompt;

modifying a texture of the object depicted in the image using the artificial texture that has been generated based on the prompt;

accessing a fashion item template based on the prompt;

providing the fashion item template to the machine learning model to populate the fashion item template with the artificial texture; and

overlaying the fashion item template populated with the artificial texture on the object depicted in the image.

2 . The method of claim 1 , wherein the prompt comprises a textual description of a fashion item worn by the object depicted in the image.

3 . The method of claim 1 , further comprising:

processing the prompt using a generative machine learning model to generate the artificial texture.

4 . The method of claim 1 , further comprising:

receiving input from the interaction client that specifies one or more parameters of the prompt.

5 . The method of claim 1 , further comprising:

processing the image to generate a segmentation of a fashion item worn by the object depicted in the image.

6 . The method of claim 5 , wherein the fashion item worn by the object is replaced with the artificial texture based on the segmentation of the fashion item.

7 . The method of claim 1 , further comprising:

continuously monitoring speech input received from a user of the interaction client; and

generating the prompt in response to detecting one or more keywords in the speech input.

8 . The method of claim 7 , wherein the one or more keywords correspond to a fashion item depicted in the image and one or more attributes.

9 . The method of claim 8 , wherein the one or more attributes comprise a fashion item style, further comprising:

adding to the prompt an identification of the fashion item and a target style corresponding to the fashion item style, wherein the artificial texture comprises an artificial fashion item in the target style.

10 . The method of claim 9 , further comprising:

identifying one or more preferences of the user based on the interaction data stored in the multimodal memory;

adding to the prompt an indication of the one or more preferences, wherein the artificial texture comprises a representation of the one or more preferences.

11 . The method of claim 10 , wherein the one or more preferences correspond to a sporting event associated with the user, wherein the fashion item style corresponds to a jersey associated with the sporting event, and wherein the one or more preferences comprise a team associated with the user and the representation comprises a logo of the team.

12 . The method of claim 1 , further comprising:

generating, by the machine learning model, an annotation representing the prompt; and

overlaying the annotation on the image that comprises the object with the modified texture.

13 . The method of claim 1 , further comprising:

detecting ambiguity in speech input received from a user of the interaction client; and

resolving the ambiguity based on the multimodal memory, wherein the prompt is generated in response to resolving the ambiguity in the speech input.

14 . The method of claim 1 , wherein the fashion item template is generated using a segmentation machine learning model that is applied to the object depicted in the image.

15 . The method of claim 14 , wherein the segmentation machine learning model is selected from a plurality of segmentation machine learning models based on keywords in the prompt.

16 . The method of claim 1 , wherein the fashion item template comprises a combination of multiple fashion item templates for different parts of a body, further comprising providing an option for a user to purchase a real-world fashion item corresponding to the fashion item template populated with the artificial texture.

17 . The method of claim 16 , further comprising:

providing a first of the multiple fashion item templates to the machine learning model to populate the first fashion item template with a first artificial texture;

providing a second of the multiple fashion item templates to the machine learning model to populate the second fashion item template with a second artificial texture; and

overlaying the first and second fashion item templates populated with the first and second artificial textures on the object depicted in the image.

18 . A system comprising:

at least one processor; and

at least one memory component having instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:

storing, in a multimodal memory, interaction data representing use of one or more interaction functions including data in different modalities;

detecting an object depicted in an image captured by an interaction client;

generating, by a machine learning model, a prompt based on the object depicted in the image and the interaction data in the multimodal memory;

generating an artificial texture based on the prompt;

modifying a texture of the object depicted in the image using the artificial texture that has been generated based on the prompt;

accessing a fashion item template based on the prompt;

providing the fashion item template to the machine learning model to populate the fashion item template with the artificial texture; and

overlaying the fashion item template populated with the artificial texture on the object depicted in the image.

19 . A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:

storing, in a multimodal memory, interaction data representing use of one or more interaction functions including data in different modalities;

detecting an object depicted in an image captured by an interaction client;

generating, by a machine learning model, a prompt based on the object depicted in the image and the interaction data in the multimodal memory;

generating an artificial texture based on the prompt;

modifying a texture of the object depicted in the image using the artificial texture that has been generated based on the prompt;

accessing a fashion item template based on the prompt;

providing the fashion item template to the machine learning model to populate the fashion item template with the artificial texture; and

overlaying the fashion item template populated with the artificial texture on the object depicted in the image.

Assignments (1)

Continuity (2)

References Cited (400)

Cited By (1)

US 12,646,266