IP Library Granted Patent US 11,076,111
Granted Patent B1
US 11,076,111 · App. 16/682,994 · Granted Jul 27, 2021

Smart color-based background replacement

Inventors: Kevin M. Ni (San Francisco, CA); Jackson Gardner (Oakland, CA); Charles Bradford (Irvine, CA); Sean Stanek (Irvine, CA); Ross Alan Cameron Gardner (Irvine, CA); Tyler McDowall (Laguna Hills, CA); Phillip Carter (Lake Forest, CA); Furqan Rydhan (San Francisco, CA); Florian Patrick Nierhaus (San Francisco, CA); Sang Jin Lee (San Francisco, CA)
Assignee: Twitch Interactive, Inc.
H04N5/272G06K9/6215G06K9/6218G06T7/90G06T2207/10016G06T2207/10024G06T2207/20021G06T2207/20072
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,076,111
App. No.
16/682,994
Granted
Jul 27, 2021
Kind
B1
Abstract

A streaming application setup assistant may receive an image captured by a camera, the image capturing a physical environment that is to be part of a live video stream. A plurality of pixels may be selected from the image. The plurality of pixels may be grouped, based on a pixel color value for each pixel, into a plurality of pixel groups. A key pixel color value may be calculated that is associated with an average pixel color value of pixels in a largest pixel group of the plurality of pixel groups. A similarity color range that encompasses a threshold percentage of the pixels in the largest pixel group may be identified based on the key pixel color value. The similarity color range may be utilized to configure settings for replacement of the background in the live video stream.

Claims (48)

1. A computing system comprising:

one or more processors; and

one or more memories having stored therein computing instructions that, upon execution by the one or more processors, cause the computing system to perform operations comprising:

receiving, by a streaming application setup assistant, an image captured by a camera, the image capturing a physical environment that is to be part of a video stream;

determining a similarity color range of a background of the image by:

selecting a plurality of pixels from the image, wherein the plurality of pixels represents a proper subset of all pixels included in the image;

determining a pixel color value for each pixel included in the plurality of pixels;

plotting the pixel color value for each pixel on a color value graph;

grouping, based on the pixel color value for each pixel, the plurality of pixels into a plurality of pixel groups;

selecting a largest pixel group of the plurality of pixel groups;

calculating a key pixel color value that is associated with an average pixel color value of pixels in the largest pixel group; and

identifying, based on a first threshold percentage of pixels in the largest pixel group, the similarity color range, wherein the similarity color range is an area formed around the key pixel color value, within the color value graph, that encompasses the first threshold percentage of the pixels in the largest pixel group, wherein the similarity color range includes pixel values that are within the area; and

utilizing the similarity color range to configure settings for replacement of the background in the video stream.

2. The computing system of claim 1 , wherein the image includes a first portion, and a second portion, and wherein the plurality of pixels are selected from the first portion and not the second portion.

3. The computing system of claim 2 , wherein the first portion of the image includes a right portion of the image and a left portion of the image and the second portion of the image includes a center portion of the image.

4. The computing system of claim 1 , wherein each pixel of the plurality of pixels is substantially equidistant from each of at least two other pixels in the plurality of pixels.

5. A computer-implemented method comprising:

receiving, by a streaming application setup assistant, an image captured by a camera, the image capturing a physical environment that is to be part of a live video stream;

determining a similarity color range of a background of the image by:

determining a pixel color value for each pixel included in a plurality of pixels within the image;

grouping, based on the pixel color value for each pixel, the plurality of pixels into a plurality of pixel groups;

selecting a largest pixel group of the plurality of pixel groups;

calculating a key pixel color value based on pixel color values of pixels in the largest pixel group; and

identifying, based on a first threshold percentage of pixels in the largest pixel group, the similarity color range, wherein the similarity color range is an area formed around the key pixel color value, within a color value graph, that encompasses the first threshold percentage of the pixels in the largest pixel group, wherein the similarity color range includes pixel values that are within the area; and

utilizing the similarity color range to configure settings for replacement of the background in the live video stream.

6. The computer-implemented method of claim 5 , wherein the image includes a first portion, and a second portion, and wherein the plurality of pixels are selected from the first portion and not the second portion.

7. The computer-implemented method of claim 6 , wherein the first portion of the image includes a right portion of the image and a left portion of the image and the second portion of the image includes a center portion of the image.

8. The computer-implemented method of claim 6 , further comprising selecting the plurality of pixels from the image based on an equal distribution of pixels in the first portion of the image.

9. The computer-implemented method of claim 5 , wherein the key pixel color value is calculated based on an average of the pixel color values in the largest pixel group.

10. The computer-implemented method of claim 5 , wherein the grouping comprises dividing a graph of pixel color values into a plurality of areas representing the plurality of pixel groups.

11. The computer-implemented method of claim 5 , wherein the grouping is performed based on a clustering algorithm.

12. The computer-implemented method of claim 5 , wherein the pixel color value for a respective pixel of the plurality of pixels is independent of a luminance value for the respective pixel.

13. The computer-implemented method of claim 12 , wherein the pixel color value for the respective pixel comprises blue chrominance (U) and red chrominance (V) values of a luminance-blue chrominance-red chrominance (YUV) color model.

14. The computer-implemented method of claim 5 , wherein the area is associated with a Euclidean distance from the key pixel color value.

15. The computer-implemented method of claim 5 , further comprising performing the replacement of the background based, at least in part, on a determination that the largest pixel group includes at least a second threshold percentage of the plurality of pixels in all the plurality of pixel groups combined.

16. One or more non-transitory computer-readable storage media having stored thereon computing instructions that, upon execution by a computing device, cause the computing device to perform operations comprising:

receiving, by a streaming application setup assistant, an image captured by a camera, the image capturing a physical environment that is to be part of a live video stream;

determining a similarity color range of a background of the image by:

determining a pixel color value for each pixel included in a plurality of pixels within the image;

grouping, based on the pixel color value for each pixel, the plurality of pixels into a plurality of pixel groups;

selecting a largest pixel group of the plurality of pixel groups;

calculating a key pixel color value that based on pixel color values of pixels in the largest pixel group; and

identifying, based on a first threshold percentage of pixels in the largest pixel group, the similarity color range, wherein the similarity color range is an area formed around the key pixel color value, within a color value graph, that encompasses the first threshold percentage of the pixels in the largest pixel group, wherein the similarity color range includes pixel values that are within the area; and

utilizing the similarity color range to configure settings for replacement of the background in the live video stream.

17. The one or more non-transitory computer-readable storage media of claim 16 , wherein the grouping comprises dividing a graph of pixel color values into a plurality of areas representing the plurality of pixel groups.

18. The one or more non-transitory computer-readable storage media of claim 16 , wherein the pixel color value for a respective pixel of the plurality of pixels is independent of a luminance value for the respective pixel.

19. The one or more non-transitory computer-readable storage media of claim 18 , wherein the pixel color value for the respective pixel comprises blue chrominance (U) and red chrominance (V) values of a luminance-blue chrominance-red chrominance (YUV) color model.

20. The one or more non-transitory computer-readable storage media of claim 16 , wherein the area is associated with a Euclidean distance from the key pixel color value.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 14, 2021
From: NI, KEVIN M.; GARDNER, JACKSON; BRADFORD, CHARLES; STANEK, SEAN; GARDNER, ROSS ALAN CAMERON; MCDOWALL, TYLER; CARTER, PHILLIP; NIERHAUS, FLORIAN PATRICK; LEE, SANG JIN
To: TWITCH INTERACTIVE, INC.
Reel/Frame 056536/0908 →
Cited By (3)
US 12,340,392 US 12,389,080 US 12,598,241