Granted Patent B2

US 12,205,295 · App. 18/243,444 · Granted Jan 21, 2025

Inventors: Gal Dudovitch (Tel Aviv, IL); Peleg Harel (Ramat Gan, IL); Chia-Hao Hsieh (Los Angeles, CA); Sergei Korolev (Marina del Rey, CA); Ma'ayan Mishin Shuvi (Givatayim, IL)

Assignee: Snap Inc.

G06T7/11G06F18/214G06F18/217G06T5/70G06T11/00G06V40/10G06T2207/10016G06T2207/20081G06T2207/20084G06T2207/30196

Abstract

Claims (52)

1. A method comprising:

accessing a video comprising a plurality of images received prior to an image;

predicting, by a first deep neural network based on the plurality of images of the video received prior to the image, a segmentation of a body depicted in the image;

comparing predicted one or more segmentations of bodies provided by a second deep neural network with the segmentation of the body predicted by the first deep neural network; and

smoothing the segmentation of the body depicted in the image based on comparing the predicted one or more segmentations of bodies with the segmentation of the body.

2. The method of claim 1 , further comprising:

generating the segmentation of the body based on the image; and

applying one or more visual effects to the image based on the smoothed segmentation.

3. The method of claim 1 , further comprising training the first deep neural network by performing operations comprising:

receiving training data comprising a plurality of training images and ground truth segmentations for each of the plurality of training images;

applying the first deep neural network to a first training image of the plurality of training images to estimate a segmentation of a given body depicted in the first training image;

computing a deviation between the estimated segmentation and the ground truth segmentation associated with the first training image; and

updating parameters of the first deep neural network based on the computed deviation.

4. The method of claim 3 , wherein the plurality of training images comprises ground truth skeletal key points of one or more bodies depicted in the respective training images.

5. The method of claim 3 , wherein the first deep neural network estimates skeletal key points of the given body depicted in the first training image.

6. The method of claim 3 , further comprising updating parameters of the first deep neural network based on a deviation between estimated skeletal key points and ground truth skeletal key points.

7. The method of claim 3 , wherein the plurality of training images comprises a plurality of image resolutions, further comprising generating a plurality of segmentation models based on the first deep neural network, a first of the plurality of segmentation models being trained based on training images having a first of the plurality of image resolutions, a second of the plurality of segmentation models being trained based on training images having a second of the plurality of image resolutions.

8. The method of claim 3 , wherein the plurality of training images comprises a plurality of labeled and unlabeled image and video data.

9. The method of claim 3 , wherein the plurality of training images comprises e a depiction of a whole body of a particular user, an image that lacks a depiction of any user, a depiction of a plurality of users, and depictions of users at different distances from an image capture device.

10. The method of claim 1 , further comprising training the second deep neural network by performing operations comprising:

receiving training data comprising a plurality of training videos and ground truth segmentations for each of the plurality of training videos;

applying the second deep neural network to a first training video of the plurality of training videos to predict a segmentation of the body in a frame subsequent to the first training video;

computing a deviation between the predicted segmentation of the body and the ground truth segmentation of the body depicted in the frame subsequent to the first training video; and

updating parameters of the second deep neural network based on the computed deviation.

11. The method of claim 1 , further comprising applying one or more visual effects to the image based on a segmentation border associated with the smoothed segmentation.

12. The method of claim 1 , further comprising:

determining one or more device capabilities of a device used to capture the image; and

selecting a segmentation model to generate the segmentation based on the one or more device capabilities.

13. The method of claim 1 , further comprising applying a guided filter to improve segmentation quality of portions of the smoothed segmentations that are within a specified number of pixels of edges of the smoothed segmentation.

14. The method of claim 1 , further comprising replacing a background of the image with a different background or replacing portions of body depicted in the image with different visual elements.

15. A system comprising:

at least one processor; and

a memory component having instructions stored thereon, when executed by the at least one processor, causes the at least one processor to perform operations comprising:

accessing a video comprising a plurality of images received prior to an image;

predicting, by a first deep neural network based on the plurality of images of the video received prior to the image, a segmentation of a body depicted in the image;

comparing predicted one or more segmentations of bodies provided by a second deep neural network with the segmentation of the body predicted by the first deep neural network; and

smoothing the segmentation of the body depicted in the image based on comparing the predicted one or more segmentations of bodies with the segmentation of the body.

16. The system of claim 15 , the operations further comprising:

generating the segmentation of the body based on the image; and

applying one or more visual effects to the image based on the smoothed segmentation.

17. The system of claim 15 , the operations further comprising training the first deep neural network by performing operations comprising:

receiving training data comprising a plurality of training images and ground truth segmentations for each of the plurality of training images;

applying the first deep neural network to a first training image of the plurality of training images to estimate a segmentation of a given body depicted in the first training image;

computing a deviation between the estimated segmentation and the ground truth segmentation associated with the first training image; and

updating parameters of the first deep neural network based on the computed deviation.

18. The system of claim 17 , wherein the plurality of training images comprises ground truth skeletal key points of one or more bodies depicted in the respective training images.

19. The system of claim 17 , wherein the first deep neural network estimates skeletal key points of the given body depicted in the first training image.

20. A non-transitory computer-readable storage medium having stored thereon, instructions when executed by at least one processor, causes the at least one processor to perform operations comprising:

accessing a video comprising a plurality of images received prior to an image;

predicting, by a first deep neural network based on the plurality of images of the video received prior to the image, a segmentation of a body depicted in the image;

comparing predicted one or more segmentations of bodies provided by a second deep neural network with the segmentation of the body predicted by the first deep neural network; and

smoothing the segmentation of the body depicted in the image based on comparing the predicted one or more segmentations of bodies with the segmentation of the body.

Assignments (1)

Continuity (2)

References Cited (392)