Granted Patent B2

US 12,260,450 · App. 18/505,420 · Granted Mar 25, 2025

Inventors: Itamar Berger (Hod Hasharon, IL); Gal Dudovitch (Tel Aviv, IL); Ma'ayan Mishin Shuvi (Givatayim, IL)

Assignee: SNAP INC.

G06Q30/0643G06N3/08G06Q30/0271G06Q30/0276G06V40/103G06V40/161

Abstract

Claims (103)

1. A method comprising:

selecting a first product type as a target product type in response to determining that a first pose corresponds to a whole first body pose in which a whole-body first type of fashion item is depicted in a first image;

selecting a second product type as the target product type in response to determining that the first pose corresponds to a partial body pose in which a second type of fashion item comprising a partial body fashion item is depicted in the first image;

searching, based on the target product type and the first type of fashion item, a plurality of products to identify a first product that corresponds to the target product type matching the first pose of a first person depicted in the first image and that includes one or more attributes associated with the first type of fashion item;

applying a trained neural network to the first image that has been previously stored by a first device to generate a segmentation to distinguish a portion of the first person depicted in the first image from a background depicted in the first image, the trained neural network being trained by performing operations comprising:

receiving training data comprising a plurality of training monocular images and ground truth segmentations for each of the plurality of training monocular images;

applying a neural network to a first training monocular image of the plurality of training monocular images to estimate a segmentation of a garment worn by a given person depicted in the first training monocular image;

computing a deviation between the estimated segmentation and the ground truth segmentations associated with the first training monocular image;

updating parameters of the neural network based on the computed deviation; and

in response to applying the trained neural network to the first image to generate the segmentation, modifying the first image to generate a content item that depicts the first person wearing the first product in which the first product placed on the portion of the first person in a manner that blends with the background of the first image, the modifying of the first image to generate the content item comprising:

calculating characteristic points for a set of elements of the first person to generate a mesh based on the calculated characteristic points;

generating one or more areas on the mesh of the first person;

aligning the one or more areas of the first person with one or more elements of a first augmented reality representation of the first product with a position; and

modifying one or more visual properties of the one or more areas to cause the first device to display the first augmented reality representation within the first image at an individual display position relative to the display position of the first person.

2. The method of claim 1 , further comprising:

accessing content previously received by an application server from a first device associated with the first person;

processing the content to identify the first image that depicts the first person wearing the first type of fashion item that has been previously stored by the first device; and

without receiving a request from the first person to view an advertisement, during a content browsing session being accessed by the first device, causing, to be displayed, the content item that depicts the first person wearing the first product blended with the background of the first image that has been previously stored by the first device.

3. The method of claim 1 , further comprising visually distinguishing a portion of the first image that corresponds to the first product from other portions of the first image to identify the first product being advertised, the visually distinguishing comprising displaying a highlight or border region around the portion of the first image corresponding to the first product.

4. The method of claim 1 , further comprising:

displaying a first video in a sequence of videos;

after the first video finishes being displayed, obtaining the content item that depicts the first person wearing the first product that has been previously generated;

automatically presenting the content item for a specified period of time; and

after the content item that depicts the first person wearing the first product has been presented for the specified period of time, automatically displaying a second video in a sequence of videos.

5. The method of claim 1 , further comprising:

storing a database that includes a plurality of matching outfit pairs, each of the plurality of matching outfit pairs comprising a given product of the plurality of products and an associated set of attributes that match the given product; and

searching the database to determine that the set of attributes associated with the first product includes at least one of the one or more attributes of the first type of fashion item.

6. The method of claim 1 , further comprising:

determining that the first image depicts a first portion and a second portion of a body of the first person, wherein the first type of fashion item is depicted in the first portion of the body of the first person;

wherein a second fashion item is depicted in the second portion of the body of the first person; and

generating the content item by replacing the second fashion item with the first product in the second portion of the body of the first person while maintaining a depiction of the first type of fashion item on the first portion of the body of the first person.

7. The method of claim 1 , further comprising:

receiving a set of products associated with advertisements;

generating a set of image selection criteria based on the set of products; and

searching content based on the set of image selection criteria to identify a plurality of images, wherein the first image is selected from the plurality of images.

8. The method of claim 7 , wherein the set of image selection criteria comprises a target pose and a target type of fashion item; and wherein the set of image selection criteria comprises an image quality criterion.

9. The method of claim 7 , further comprising:

modifying the plurality of images to generate a plurality of advertisements that depict the first person wearing the first product, the plurality of advertisements including the advertisement displayed on the first device;

after causing the content item to be displayed automatically on the first device, receiving a request for additional views of the first person wearing the first product; and

in response to receiving the request for the additional views, causing another advertisement of the plurality of advertisements that depicts the first person wearing the first product to be displayed on the first device.

10. The method of claim 7 , further comprising:

selecting an occlusion pattern for the first product, the occlusion pattern controlling which subset of pixels of a real-world garment worn by the first person is overlapped by a subset of pixels of the first product, wherein the subset of pixels of the real-world garment that are within an overlap region are replaced with the subset of pixels of the first product, the real-world garment corresponding to a first type of fashion item worn on a first body part and the first product corresponding to a second type of fashion item worn on a second body part.

11. The method of claim 7 , wherein the set of products include makeup; and wherein the set of image selection criteria comprises a portrait view of a face, further comprising:

searching the content to identify the plurality of images that depicts the portrait view of the face of the first person.

12. The method of claim 1 , further comprising:

obtaining a plurality of images; and

generating a first set of target images by:

applying one or more object recognition processes to the plurality of images to identify more than one human face in a subset of images;

removing the subset of images from the plurality of images in response to applying the one or more object recognition processes for detecting more than one human face in the subset of images; and

applying facial recognition to the plurality of images from which the subset of images has been removed to select for inclusion in the first set of target images any image that depicts a known face of the first person.

13. The method of claim 1 , further comprising:

obtaining a plurality of images;

generating a first set of target images by:

identifying from the plurality of images one or more images that depicts more than one person;

identifying a dominant person in the one or more images, the dominant person having facial characteristics that match facial characteristics of the first person or being depicted more prominently than other persons; and

modifying a fashion item worn by the dominant person in the one or more images to generate the content item that depicts the dominant person wearing the first product.

14. The method of claim 1 , wherein the content item that depicts the first person wearing the first product is presented among a set of media items in an interactive mosaic display.

15. The method of claim 1 , wherein the first product type comprises a dress that is selected in response to detecting a dress being depicted as being worn by the first person in the first image, and wherein the first product type comprises a shirt that is selected in response to detecting a shirt being depicted as being worn by the first person in the first image.

16. The method of claim 1 , further comprising:

determining a fashion profile associated with the first person;

searching, based on the fashion profile associated with the first person, the plurality of products to identify a set of products that match the fashion profile; and

searching, based on the first pose of the first person and the first type of fashion item, the set of products to identify the first product that corresponds to the first pose of the first person depicted in the first image and that includes one or more attributes associated with the first type of fashion item.

17. The method of claim 16 , further comprising:

identifying a plurality of images that depict the first person, determining a plurality of types of fashion items worn by the first person across the plurality of images;

computing a quantity for each of the plurality of types representing a number of instances of each of the plurality of types across the plurality of images;

comparing the quantity of each of the plurality of types to a threshold value; and

in response to determining that a first of the plurality of types transgresses the threshold value, adding the first of the plurality of types to the fashion profile associated with the first person.

18. The method of claim 16 , further comprising:

accessing one or more messages the first person exchanged with another user via an application server;

determining one or more types of fashion items mentioned in the one or more messages, the one or more messages being associated with an event that occurs at a particular time;

adding the one or more types of fashion items mentioned in the one or more messages to the fashion profile associated with the first person;

determining that a current time is after the particular time at which the event occurs; and

in response to determining that the current time is after the particular time at which the event occurs, removing the one or more types of fashion items mentioned in the one or more messages from the fashion profile associated with the first person.

19. A system comprising:

at least one processor; and

a memory component having instructions stored thereon, when executed by the at least one processor, causes the at least one processor to perform operations comprising:

selecting a first product type as a target product type in response to determining that a first pose corresponds to a whole first body pose in which a whole-body first type of fashion item is depicted in a first image;

selecting a second product type as the target product type in response to determining that the first pose corresponds to a partial body pose in which a second type of fashion item comprising a partial body fashion item is depicted in the first image;

searching, based on the target product type and the first type of fashion item, a plurality of products to identify a first product that corresponds to the target product type matching the first pose of a first person depicted in the first image and that includes one or more attributes associated with the first type of fashion item;

applying a trained neural network to the first image that has been previously stored by a first device to generate a segmentation to distinguish a portion of the first person depicted in the first image from a background depicted in the first image, the trained neural network being trained by performing operations comprising:

receiving training data comprising a plurality of training monocular images and ground truth segmentations for each of the plurality of training monocular images;

applying a neural network to a first training monocular image of the plurality of training monocular images to estimate a segmentation of a garment worn by a given person depicted in the first training monocular image;

computing a deviation between the estimated segmentation and the ground truth segmentations associated with the first training monocular image;

updating parameters of the neural network based on the computed deviation; and

in response to applying the trained neural network to the first image to generate the segmentation, modifying the first image to generate a content item that depicts the first person wearing the first product in which the first product placed on the portion of the first person in a manner that blends with the background of the first image, the modifying of the first image to generate the content item comprising:

calculating characteristic points for a set of elements of the first person to generate a mesh based on the calculated characteristic points;

generating one or more areas on the mesh of the first person;

aligning the one or more areas of the first person with one or more elements of a first augmented reality representation of the first product with a position; and

modifying one or more visual properties of the one or more areas to cause the first device to display the first augmented reality representation within the first image at an individual display position relative to the display position of the first person.

20. A non-transitory computer-readable storage medium having stored thereon, instructions when executed by at least one processor, causes the at least one processor to perform operations comprising:

selecting a first product type as a target product type in response to determining that a first pose corresponds to a whole first body pose in which a whole-body first type of fashion item is depicted in a first image;

selecting a second product type as the target product type in response to determining that the first pose corresponds to a partial body pose in which a second type of fashion item comprising a partial body fashion item is depicted in the first image;

searching, based on the target product type and the first type of fashion item, a plurality of products to identify a first product that corresponds to the target product type matching the first pose of a first person depicted in the first image and that includes one or more attributes associated with the first type of fashion item;

applying a trained neural network to the first image that has been previously stored by a first device to generate a segmentation to distinguish a portion of the first person depicted in the first image from a background depicted in the first image, the trained neural network being trained by performing operations comprising:

receiving training data comprising a plurality of training monocular images and ground truth segmentations for each of the plurality of training monocular images;

applying a neural network to a first training monocular image of the plurality of training monocular images to estimate a segmentation of a garment worn by a given person depicted in the first training monocular image;

computing a deviation between the estimated segmentation and the ground truth segmentations associated with the first training monocular image;

updating parameters of the neural network based on the computed deviation; and

in response to applying the trained neural network to the first image to generate the segmentation, modifying the first image to generate a content item that depicts the first person wearing the first product in which the first product placed on the portion of the first person in a manner that blends with the background of the first image, the modifying of the first image to generate the content item comprising:

calculating characteristic points for a set of elements of the first person to generate a mesh based on the calculated characteristic points;

generating one or more areas on the mesh of the first person;

aligning the one or more areas of the first person with one or more elements of a first augmented reality representation of the first product with a position; and

modifying one or more visual properties of the one or more areas to cause the first device to display the first augmented reality representation within the first image at an individual display position relative to the display position of the first person.

Assignments (1)

Continuity (2)

References Cited (207)

Cited By (1)

US 12,394,239