Granted Patent B2

US 12,462,507 · App. 18/098,799 · Granted Nov 4, 2025

Inventors: Itamar Berger (Hod Hasharon, IL); Gal Dudovitch (Tel Aviv, IL); Gal Sasson (Kibbutz Ayyelet Hashahar, IL); Ma'ayan Mishin Shuvi (Givatayim, IL); Matan Zohar (Rishon LeZion, IL)

Assignee: Snap Inc.

G06T19/20G06T7/10G06T17/00G06T19/006G06T2207/20084

Abstract

Claims (56)

1 . A method comprising:

capturing, by a camera of a user device, an image of a person wearing a fashion item;

estimating an angle of one or more pixels of the fashion item depicted in a region of the image relative to the camera; and

applying one or more augmented reality (AR) elements to the region of the image, captured by the camera, that includes the depiction of the fashion item in the image based on the estimated angle of the one or more pixels.

2 . The method of claim 1 , wherein before estimating the angle of one or more pixels in the image, the method comprises:

receiving the image that includes a depiction of a person wearing a fashion item;

generating a segmentation of the fashion item worn by the person depicted in the image; and

extracting a portion of the image corresponding to the segmentation of the fashion item, the angle being estimated based on the extracted portion of the image.

3 . The method of claim 1 , wherein applying the one or more AR elements comprises:

determining that light is being focused on the fashion item from a first direction based on the estimated angle; and

modifying pixel values of the fashion item to re-focus the light on the fashion item from a second direction based on the estimated angle.

4 . The method of claim 3 , further comprising:

determining that a first pixel of the image is pointing towards a given direction relative to the camera; and

modifying the first pixel to render a reflection of re-focused light from the second direction towards the given direction.

5 . The method of claim 1 , wherein applying the one or more AR elements comprises:

selecting an AR material for the fashion item; and

replacing a material of the fashion item with the AR material based on the estimated angle.

6 . The method of claim 5 , further comprising:

determining a light reflection or absorption property of the AR material; and

for each of the one or more pixels, computing a new pixel value based on the light reflection or absorption property of the AR material and the estimated angle of the pixel.

7 . The method of claim 5 , further comprising applying different AR shades to the fashion item for each different type of the AR material that is selected.

8 . The method of claim 5 , further comprising:

displaying a list of different AR materials; and

receiving a selection of the AR material from the list.

9 . The method of claim 5 , wherein the AR material comprises gold, mirror, water, or slime.

10 . The method of claim 1 , wherein applying the one or more AR elements comprises applying artificial light to the fashion item based on the estimated angle.

11 . The method of claim 1 , further comprising:

displaying the one or more AR elements on a first portion of the fashion item in a first frame of a video, wherein the fashion item is positioned at a first location in the first frame;

determining that the fashion item has moved from the first location to a second location in a second frame of the video, the first portion of the fashion item being moved to a new location in the second frame of the video; and

updating a display position of the one or more AR elements in the second frame to maintain the display of the one or more AR elements on the first portion of the fashion item.

12 . The method of claim 1 , further comprising:

determining that the fashion item depicted in the image has been rotated by a specified amount; and

rotating the one or more AR elements by the specified amount in response to determining that the fashion item has been rotated by the specified amount.

13 . The method of claim 1 , wherein the estimated angle is computed relative to a surface normal of the camera.

14 . The method of claim 1 , wherein estimating the angle comprises:

applying a neural network to the image, the neural network being trained to establish a relationship between image portions depicting different orientations of real-world items and pixel directions of the real-world items relative to surface normals of cameras used to capture the image portions.

15 . The method of claim 1 , wherein estimating the angle comprises:

applying a neural network to a portion of the image, the neural network being trained to establish a relationship between image portions depicting reflections of light from different directions on real-world items and pixel directions of the real-world items relative to surface normals of cameras used to capture the image portions.

16 . The method of claim 1 , further comprising:

receiving a request to apply AR light to the fashion item from a new direction; and

training a neural network to estimate lighting conditions for each pixel when the AR light is applied from the new direction.

17 . The method of claim 16 , wherein the neural network is trained to establish a relationship between image portions depicting focusing of light on real-world items from different directions and lighting conditions of pixels of the real-world items.

18 . A system comprising:

at least one processor of a user device; and

a memory component having instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:

capturing by a camera of the user device, an image of a person wearing a fashion item;

estimating an angle of one or more pixels of the fashion item depicted in a region of the image relative to the camera; and

applying one or more augmented reality (AR) elements to the region of the image, captured by the camera, that includes the depiction of the fashion item in the image based on the estimated angle of the one or more pixels.

19 . The system of claim 18 , wherein before estimating the angle of one or more pixels in the image, the operations comprising:

receiving the image that includes a depiction of a person wearing a fashion item;

generating a segmentation of the fashion item worn by the person depicted in the image; and

extracting a portion of the image corresponding to the segmentation of the fashion item, the angle being estimated based on the extracted portion of the image.

20 . A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by at least one processor of a user device, cause the at least one processor to perform operations comprising:

capturing, by a camera of the user device, an image of a person wearing a fashion item;

estimating an angle of one or more pixels of the fashion item depicted in a region of the image relative to the camera; and

applying one or more augmented reality (AR) elements to the region of the image, captured by the camera, that includes the depiction of the fashion item in the image based on the estimated angle of the one or more pixels.

Assignments (2)

Continuity (2)

References Cited (400)

Cited By (1)

US 12,670,643