Granted Patent B2

US 12,354,211 · App. 18/179,920 · Granted Jul 8, 2025

Inventors: Ron Weitzman (Tel Aviv, IL); Oleksandr Pyshchenko (Los Angeles, CA); Raz Perlman (Rehovot, IL); Vivian Su (Maynard, MA); Almog Mizrahi (Tel Aviv, IL)

Assignee: SNAP INC.

G06T15/06G06T19/006

Abstract

Claims (75)

1. A method comprising:

accessing, by a mobile device, a video depicting a first object;

obtaining, by the mobile device, a three-dimensional (3D) model of the first object,

in response to receiving input that activates a ray tracing process, applying, by the mobile device, the ray tracing process to the 3D model of the first object to estimate an optical effect on a portion of the first object relative to a second object that is depicted in the video;

modifying a visual property of the portion of the first object based on the optical effect relative to the second object;

detecting a first region of the second object in a first frame of the video;

storing a texture corresponding to the first region that is depicted in the first frame;

determining that the first region of the second object is no longer visible in a second frame of the video that is subsequent to the first frame; and

modifying the visual property of the portion of the first object depicted in the second frame using the texture of the first region that is no longer visible in the second frame.

2. The method of claim 1 , further comprising:

in response to receiving the input to activate the ray tracing process, animating, over a plurality of video frames subsequent, the modifying of the visual property of the portion of the first object based on the optical effect relative to the second object.

3. The method of claim 1 , wherein the first object is a first augmented reality (AR) object, and wherein the second object is a second AR object.

4. The method of claim 1 , wherein the optical effect comprises at least one of reflection, refraction, soft shadows, scattering, depth of field, motion blur, ambient occlusion, or dispersion.

5. The method of claim 1 , further comprising:

applying a first trained machine learning model to the first object to generate a segmentation of the first object; and

applying a second trained machine learning model to the segmentation to generate the 3D model of the first object, the 3D model of the first object comprising a transparent version of the first object.

6. The method of claim 1 , further comprising:

in response to applying the ray tracing process, identifying a region of the second object that is intersected by one or more rays associated with the portion of the first object;

retrieving a light attribute of the region of the second object; and

blending the light attribute of the region of the second object into a light attribute of the portion of the first object.

7. The method of claim 6 , further comprising:

receiving configuration information indicating a level of quality for the optical effect.

8. The method of claim 7 , further comprising:

determining that the level of quality is greater than a threshold quality level; and

in response to determining that the level of quality level is greater than the threshold quality level, generating a plurality of rays as the one or more rays in association with a single pixel of the portion of the first object.

9. The method of claim 7 , further comprising:

determining that the level of quality is less than a threshold quality level; and

in response to determining that the level of quality is less than the threshold quality level, generating a single ray as the one or more rays in association with a single pixel of the portion of the first object.

10. The method of claim 6 , wherein the light attribute of the region of the second object comprises a first intensity, and wherein the light attribute of the portion of the first object comprises a second intensity.

11. The method of claim 1 , further comprising:

selecting between generating a single ray and generating a plurality of rays for the ray tracing process based on a determined level of quality associated with the optical effect indicated in configuration information.

12. The method of claim 1 , wherein the portion of the first object reflects the texture of the first region of the second object, wherein the first object comprises an augmented reality fashion item, and wherein the second object comprises a body part of a real-world person depicted in the video.

13. The method of claim 1 , further comprising:

constructing a three-dimensional (3D) model of the second object that represents textures of the second object of portions that are visible and portions that are absent from the video over a plurality of video frames.

14. The method of claim 13 , further comprising:

in response to applying the ray tracing process, identifying the first region of the second object that is intersected by one or more rays associated with the portion of the first object in the second frame;

identifying the first region in the 3D model of the second object; and

retrieving the texture of the first region from the 3D model to modify the visual property of the portion of the first object.

15. The method of claim 1 , further comprising:

applying, in a first frame of the video, a first modification to the portion of the first object in response to determining that the second object has a first light attribute;

determining that the second object has been changed to have a second light attribute in a second frame of the video; and

in response to determining that the second object has been changed to have the second light attribute, applying, in the second frame of the video, a second modification to the portion of the first object.

16. The method of claim 1 , further comprising:

displaying the first frame of the video that depicts the first object and the second object;

in response to receiving the input to activate the ray tracing process, animating, over a plurality of video frames subsequent to the first frame, the modifying of the visual property of the portion of the first object based on the optical effect relative to the second object, wherein the animating comprises:

presenting, in the second frame subsequent to the first frame, a first section of the first object comprising initial visual attributes depicted in the first frame unmodified by the optical effect while a second section of the first object comprises a modified visual attribute that is generated based on the optical effect; and

presenting, in a third frame subsequent to the second frame, the first section of the first object comprising the modified visual attributes associated with the optical effect together with the second section of the first object.

17. A system comprising:

at least one processor of a mobile device programmed to perform operations comprising:

accessing, by the mobile device, a video depicting a first object;

obtaining, by the mobile device, a three-dimensional (3D) model of the first object;

in response to receiving input that activates a ray tracing process, applying, by the mobile device, the ray tracing process to the 3D model of the first object to estimate an optical effect on a portion of the first object relative to a second object that is depicted in the video;

modifying a visual property of the portion of the first object based on the optical effect relative to the second object;

detecting a first region of the second object in a first frame of the video;

storing a texture corresponding to the first region that is depicted in the first frame;

determining that the first region of the second object is no longer visible in a second frame of the video that is subsequent to the first frame; and

modifying the visual property of the portion of the first object depicted in the second frame using the texture of the first region that is no longer visible in the second frame.

18. A non-transitory machine-readable storage medium that includes instructions that, when executed by one or more processors of a mobile device, cause the mobile device to perform operations comprising:

accessing, by the mobile device, a video depicting a first object;

obtaining, by the mobile device, a three-dimensional (3D) model of the first object;

in response to receiving input that activates a ray tracing process, applying, by the mobile device, the ray tracing process to the 3D model of the first object to estimate an optical effect on a portion of the first object relative to a second object that is depicted in the video;

modifying a visual property of the portion of the first object based on the optical effect relative to the second object;

detecting a first region of the second object in a first frame of the video;

storing a texture corresponding to the first region that is depicted in the first frame;

determining that the first region of the second object is no longer visible in a second frame of the video that is subsequent to the first frame; and

modifying the visual property of the portion of the first object depicted in the second frame using the texture of the first region that is no longer visible in the second frame.

19. The non-transitory machine-readable storage medium of claim 18 , the operations comprising:

applying, in a first frame of the video, a first modification to the portion of the first object in response to determining that the second object has a first light attribute;

determining that the second object has been changed to have a second light attribute in a second frame of the video; and

in response to determining that the second object has been changed to have the second light attribute, applying, in the second frame of the video, a second modification to the portion of the first object.

20. The non-transitory machine-readable storage medium of claim 18 , the operations comprising:

displaying the first frame of the video that depicts the first object and the second object;

in response to receiving the input to activate the ray tracing process, animating, over a plurality of video frames subsequent to the first frame, the modifying of the visual property of the portion of the first object based on the optical effect relative to the second object, wherein the animating comprises:

presenting, in the second frame subsequent to the first frame, a first section of the first object comprising initial visual attributes depicted in the first frame unmodified by the optical effect while a second section of the first object comprises a modified visual attribute that is generated based on the optical effect; and

presenting, in a third frame subsequent to the second frame, the first section of the first object comprising the modified visual attributes associated with the optical effect together with the second section of the first object.

Assignments (1)

Continuity (1)

References Cited (71)