IP Library Granted Patent US 11,961,213
Granted Patent B2
US 11,961,213 · App. 18/305,187 · Granted Apr 16, 2024

Image segmentation and modification of a video stream

Inventors: Igor Kudriashov (Saratov, RU); Fedir Poliakov (Marina Del Rey, CA); Maksim Gusarov (Marina del Rey, CA)
Assignee: Snap Inc.
G06T5/008G06T3/40G06T7/11G06T7/136G06T7/20G06T7/73G06T7/90G06T11/001G06V10/50G06V10/56G06V40/165G06V40/171G06T2207/10016G06T2207/30201G06V10/467
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,961,213
App. No.
18/305,187
Granted
Apr 16, 2024
Kind
B2
Abstract

Systems, devices, media, and methods are presented for segmenting an image of a video stream with a client device, identifying an area of interest, generating a modified area of interest within one or more image, identifying a first set of pixels and a second set of pixels, and modifying a color value for the first set of pixels.

Claims (72)

1. A computer implemented method comprising:

accessing a video stream comprising a first set of images and a second set of images;

identifying an object of interest based on a prominent position of the object of interest in the first set of images;

based on the identification, determining an approximate location of the object of interest;

identifying an area of interest within one or more images of the first set of images, the area of interest being a portion of the one or more images encompassing the approximate location of the object of interest;

generating a modified area of interest by converting pixels within the area of interest to values along a grayscale to generate a grayscale area of interest, the pixels converted to grayscale by, for each pixel, generating a grayscale value by multiplying a fixed intensity value with a quotient of a triplet of color saturation values of the pixel;

identifying a first set of pixels and a second set of pixels within the modified area of interest; and

modifying a color value for the first set of pixels within the second set of images of the video stream.

2. The method of claim 1 further comprising:

cropping one or more portions of the one or more images of the first set of images outside of the area of interest.

3. The method of claim 1 , wherein the pixels within the grayscale area of interest comprise a histogram of values within the grayscale and generating the modified area of interest further comprises:

equalizing the histogram values within the grayscale area of interest to generate an equalized area of interest.

4. The method of claim 3 , wherein generating the modified area of interest further comprises:

thresholding the equalized area of interest to generate a binarized area of interest, the first set of pixels within the binarized area of interest having a first value and the second set of pixels within the binarized area of interest having a second value distinct from the first value.

5. The method of claim 4 , wherein the second set of pixels comprises a plurality of segments interrupted by one or more pixels having the first value and generating the binarized area of interest further comprising:

identifying one or more pixels having the first value positioned between two or more of the plurality of segments having the second value; and

replacing the first value of the identified one or more pixels with the second value.

6. The method of claim 4 , wherein identifying a first set of pixels and a second set of pixels within the modified area of interest further comprises:

determining the first set of pixels have a value greater than a predetermined threshold; and

marking the first set of pixels within the second set of images of the video stream to generate a set of marked pixels.

7. The method of claim 6 , wherein marking the first set of pixels within the second set of images further comprises:

identifying locations for the first set of pixels with respect to one or more landmarks of a face depicted within the first set of images; and

identifying the color value of the first set of pixels within the first set of images.

8. The method of claim 6 , wherein modifying the color value for the first set of pixels within the second set of images further comprises:

tracking the set of marked pixels across the second set of images of the video stream; and

modifying the color value for the first set of pixels when the color value is presented proximate to the marked pixels.

9. A system for manipulating portions of a video stream, comprising:

one or more processor; and

a non-transitory machine-readable storage medium storing processor executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:

accessing, by the processor, a video stream comprising a first set of images and a second set of images;

identifying an object of interest based on a prominent position of the object of interest in the first set of images;

based on the identification, determining an approximate location of the object of interest;

identifying an area of interest within one or more images of the first set of images, the area of interest being a portion of the one or more images encompassing the approximate location of the object of interest;

generating a modified area of interest by converting pixels within the area of interest to values along a grayscale to generate a grayscale area of interest, the pixels converted to grayscale by, for each pixel, generating a grayscale value by multiplying a fixed intensity value with a quotient of a triplet of color saturation values of the pixel;

identifying a first set of pixels and a second set of pixels within the modified area of interest; and

modifying a color value for the first set of pixels within the second set of images of the video stream.

10. The system of claim 9 , wherein the pixels within the grayscale area of interest comprise a histogram of values within the grayscale and generating the modified area of interest causes the machine to perform operations comprising:

equalizing the histogram values within the grayscale area of interest to generate an equalized area of interest.

11. The system of claim 10 , wherein generating the modified area of interest causes the machine to perform operations comprising:

thresholding the equalized area of interest to generate a binarized area of interest, the first set of pixels within the binarized area of interest having a first value and the second set of pixels within the binarized area of interest having a second value distinct from the first value.

12. The system of claim 11 , wherein the second set of pixels comprises a plurality of segments interrupted by one or more pixels having the first value and generating the binarized area of interest causes the machine to perform operations comprising:

identifying one or more pixels having the first value positioned between two or more of the plurality of segments having the second value; and

replacing the first value of the identified one or more pixels with the second value.

13. The system of claim 11 , wherein identifying a first set of pixels and a second set of pixels within the modified area of interest causes the machine to perform operations comprising:

determining the first set of pixels have a value greater than a predetermined threshold; and

marking the first set of pixels within the second set of images of the video stream to generate a set of marked pixels.

14. The system of claim 13 , wherein marking the first set of pixels within the second set of images causes the machine to perform operations comprising:

identifying locations for the first set of pixels with respect to one or more landmarks of a face depicted within the first set of images; and

identifying the color value of the first set of pixels within the first set of images.

15. The system of claim 13 , wherein modifying the color value for the first set of pixels within the second set of images causes the machine to perform operations comprising:

tracking the set of marked pixels across the second set of images of the video stream; and

modifying the color value for the first set of pixels when the color value is presented proximate to the marked pixels.

16. A non-transitory machine-readable storage medium storing processor executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:

accessing, by the processor, a video stream comprising a first set of images and a second set of images;

identifying an object of interest based on a prominent position of the object of interest in the first set of images;

based on the identification, determining an approximate location of the object of interest;

identifying an area of interest within one or more images of the first set of images, the area of interest being a portion of the one or more images encompassing the approximate location of the object of interest;

generating a modified area of interest by converting pixels within the area of interest to values along a grayscale to generate a grayscale area of interest, the pixels converted to grayscale by, for each pixel, generating a grayscale value by multiplying a fixed intensity value with a quotient of a triplet of color saturation values of the pixel;

identifying a first set of pixels and a second set of pixels within the modified area of interest; and

modifying a color value for the first set of pixels within the second set of images of the video stream.

17. The non-transitory machine-readable storage medium of claim 16 , wherein the pixels within the grayscale area of interest comprise a histogram of values within the grayscale and generating the modified area of interest causes the machine to perform operations comprising:

equalizing the histogram values within the grayscale area of interest to generate an equalized area of interest.

18. The method of claim 1 further comprising:

cropping one or more portions of the one or more images of the first set of images outside of the area of interest.

19. The non-transitory machine-readable storage medium of claim 16 , wherein identifying the first set of pixels and the second set of pixels within the modified area of interest causes the machine to perform operations comprising:

determining the first set of pixels have a value within the modified area of interest greater than a predetermined threshold; and

marking the first set of pixels within the second set of images of the video stream to generate a set of marked pixels.

20. The non-transitory machine-readable storage medium of claim 19 , wherein modifying the color value for the first set of pixels within the second set of images causes the machine to perform operations comprising:

identifying locations for the first set of pixels with respect to one or more landmarks of a face depicted within the first set of images;

identifying the color value of the first set of pixels within the first set of images;

tracking the set of marked pixels across the second set of images of the video stream; and

modifying the color value for the first set of pixels when the color value is presented proximate to the marked pixels.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 23, 2024
From: KUDRIASHOV, IGOR; POLIAKOV, FEDIR; GUSAROV, MAKSIM
To: SNAPCHAT, INC.
Reel/Frame 066538/0103 →
CHANGE OF NAME Recorded Feb 23, 2024
From: SNAPCHAT, INC.
To: SNAP INC.
Reel/Frame 066538/0237 →
Continuity (5)
Continuation 17319381 · May 13, 2021
Continuation 16698463 · Nov 27, 2019
Continuation 16232737 · Dec 26, 2018
Continuation 14953726 · Nov 30, 2015
Related Publication 20230252610A1 · Aug 10, 2023