Granted Patent B2

US 10,102,423 · App. 15/199,482 · Granted Oct 16, 2018

Inventors: Victor Shaburov (Pacific Palisades, CA); Yurii Monastyrshyn (Santa Monica, CA); Oleksandr Pyshchenko (Los Angeles, CA); Sergei Kotcur (Sochi, RU)

Assignee: Snap Inc.

G06K9/00382G06T7/0048G06T7/408G06T2207/10024G06T2207/20072G06T2207/30196

Abstract

Claims (95)

1. A method, comprising:

receiving, by one or more processors, one or more images depicting at least a portion of a hand;

determining pixels within the one or more images corresponding to the portion of the hand in a predetermined portion of a field of view of an image capture device, the portion of the hand having a finger;

based on the pixels corresponding to the portion of the hand, identifying a finger position of the finger;

determining a direction of the finger based on the finger position;

based on the direction of the finger, dynamically modifying a histogram threshold for identifying pixels as corresponding to the portion of the hand; and

replacing the portion of the hand and the finger with a graphical interface element aligned with the direction of the finger.

2. The method of claim 1 , wherein identifying the finger position further comprises:

forming a convex polygon encompassing at least a part of the portion of the hand; and

identifying one or more defects within the convex polygon, a defect indicating a space between two fingers located on the portion of the hand.

3. The method of claim 1 , wherein determining the direction of the finger further comprises:

identifying a tip of the finger, the tip representing a first point for the finger;

identifying a second point on the finger, the second point spaced a distance from the first point along the finger;

generating a direction line extending between the first point and the second point; and

determining a direction of the direction line within the one or more images.

4. The method of claim 1 , wherein determining the pixels corresponding to the portion of the hand further comprises:

sampling one or more color values from one or more pixels within the portion of the field of view of the image capture device; and

determining a color profile for the hand based on the one or more color values sampled from the one or more pixels, the color profile representing a median color value of the hand.

5. The method of claim 4 , wherein sampling the one or more color values further comprises:

selecting a first pixel within the portion of the field of view;

determining that the first pixel includes a first color value within a predetermined range of color values;

selecting a second pixel within the portion of the field of view, the second pixel being spaced a distance away from the first pixel and within the portion of the field of view;

determining that the second pixel includes a second color value within the predetermined range of color values;

comparing the first color value and the second color value to determine that the second color value is within a predetermined threshold of the first color value; and

including the first color value and the second color value in the color profile for the hand.

6. The method of claim 4 , wherein determining the color profile further comprises:

generating a histogram from the one or more color values sampled from within the portion of the field of view;

removing one or more bins of the histogram associated with a number of pixels below a predetermined pixel threshold; and

including one or more bins of the histogram within the color profile associated with a number of pixels above the predetermined pixel threshold.

7. The method of claim 1 , further comprising:

generating one or more binary images by isolating the pixels corresponding to the portion of the hand by converting the pixels corresponding to the portion of the hand to a first value and remaining pixels within the field of view to a second value.

8. The method of claim 1 , wherein determining the direction of the finger further comprises:

determining a direction of the finger for each frame of a set of previous frames of the one or more images; and

combining the direction of the finger for the set of previous frames to identify an aggregate direction of the finger.

9. The method of claim 1 , wherein determining the direction of the finger further comprises:

determining a current direction of the finger from a specified corner of the field of view;

identifying a combined direction of the finger for the one or more images to indicate a previous combined direction;

determining that a change in position between the current direction and the previous combined direction exceeds a position threshold; and

selecting from a set of first direction identification operations and a set of second direction identification operations based on the change in position exceeding the position threshold.

10. The method of claim 1 , wherein the direction of the finger is represented by a vector extending between two pixels corresponding to the portion of the hand and dynamically modifying the histogram threshold further comprises:

identifying a first performance characteristic of the vector;

determining that the first performance characteristic exceeds a characteristic threshold;

based on the first performance characteristic, modifying the histogram threshold; and

based on the modification of the histogram threshold, identifying that a second performance characteristic of the vector is within the characteristic threshold.

11. The method of claim 1 , wherein determining the pixels within the one or more images corresponding to the portion of the hand identifies a first set of pixels and further comprises:

determining that a set of artifacts within the first set of pixels exceeds an artifact threshold; and

based on modifying the histogram threshold, determining a second set of pixels within the one or more images corresponding to the portion of the hand, the second set of pixels including at least a portion of the first set of pixels.

12. A system, comprising:

one or more processors; and

a processor-readable storage device storing processor-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

receiving, by the one or more processors, one or more images depicting at least a portion of a hand;

determining pixels within the one or more images corresponding to the portion of the hand in a predetermined portion of a field of view of an image capture device, the portion of the hand having a finger;

based on the pixels corresponding to the portion of the hand, identifying a finger position of the finger;

determining a direction of the finger based on the finger position;

based on the direction of the finger, dynamically modifying a histogram threshold for identifying pixels as corresponding to the portion of the hand; and

replacing the portion of the hand and the finger with a graphical interface element aligned with the direction of the finger.

13. The system of claim 12 , wherein determining the direction of the finger further comprises:

identifying a tip of the finger, the tip representing a first point for the finger;

identifying a second point on the finger, the second point spaced a distance from the first point along the finger;

generating a direction line extending between the first point and the second point; and

determining a direction of the direction line within the one or more images.

14. The system of claim 12 , wherein determining the pixels corresponding to the portion of the hand further comprises:

sampling one or more color values from one or more pixels within the portion of the field of view of the image capture device; and

determining a color profile for the hand based on the one or more color values sampled from the one or more pixels, the color profile representing a median color value of the hand.

15. The system of claim 14 , wherein determining the color profile further comprises:

generating a histogram from the one or more color values sampled from within the portion of the field of view;

removing one or more bins of the histogram associated with a number of pixels below a predetermined pixel threshold; and

including one or more bins of the histogram within the color profile associated with a number of pixels above the predetermined pixel threshold.

16. The system of claim 12 , wherein determining the direction of the finger further comprises:

determining a current direction of the finger from a specified corner of the field of view;

identifying a combined direction of the finger for the one or more images to indicate a previous combined direction;

determining that a change in position between the current direction and the previous combined direction exceeds a position threshold; and

selecting from a set of first direction identification operations and a set of second direction identification operations based on the change in position exceeding the position threshold.

17. The system of claim 12 , wherein the direction of the finger is represented by a vector extending between two pixels corresponding to the portion of the hand and dynamically modifying the histogram threshold further comprises:

identifying a first performance characteristic of the vector;

determining that the first performance characteristic exceeds a characteristic threshold;

based on the first performance characteristic, modifying the histogram threshold; and

based on the modification of the histogram threshold, identifying that a second performance characteristic of the vector is within the characteristic threshold.

18. A non-transitory processor-readable storage device storing processor-executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:

receiving one or more images depicting at least a portion of a hand;

determining pixels within the one or more images corresponding to the portion of the hand in a predetermined portion of a field of view of an image capture device, the portion of the hand having a finger;

based on the pixels corresponding to the portion of the hand, identifying a finger position of the finger;

determining a direction of the finger based on the finger position;

based on the direction of the finger, dynamically modifying a histogram threshold for identifying pixels as corresponding to the portion of the hand; and

replacing the portion of the hand and the finger with a graphical interface element aligned with the direction of the finger.

19. The non-transitory processor-readable storage device of claim 18 , wherein determining the direction of the finger further comprises:

identifying a tip of the finger, the tip representing a first point for the finger;

identifying a second point on the finger, the second point spaced a distance from the first point along the finger;

generating a direction line extending between the first point and the second point; and

determining a direction of the direction line within the one or more images.

20. The non-transitory processor-readable storage device of claim 18 , wherein determining the direction of the finger further comprises:

determining a current direction of the finger from a specified corner of the field of view;

identifying a combined direction of the finger for the one or more images to indicate a previous combined direction;

determining that a change in position between the current direction and the previous combined direction exceeds a position threshold; and

selecting from a set of first direction identification operations and a set of second direction identification operations based on the change in position exceeding the position threshold.

Continuity (1)

Cited By (112)

US 12,190,739 US 12,192,426 US 12,200,399 US 12,204,105 US 12,204,693 US 12,204,807 US 12,210,736 US 12,212,536 US 12,216,519 US 12,216,823 US 12,216,827 US 12,223,402 US 12,229,342 US 12,231,437 US 12,236,148 US 12,236,512 US 12,244,552 US 12,265,663 US 12,265,664 US 12,266,057 US 12,271,517 US 12,271,532 US 12,271,586 US 12,282,607 US 12,287,647 US 12,288,298 US 12,299,798 US 12,300,275 US 12,307,287 US 12,314,472 US 12,314,485 US 12,327,302 US 12,332,438 US 12,333,658 US 12,340,142 US 12,344,409 US 12,348,855 US 12,353,628 US 12,353,840 US 12,360,531 US 12,360,663 US 12,361,648 US 12,361,664 US 12,367,606 US 12,367,616 US 12,372,975 US 12,373,096 US 12,374,055 US 12,380,618 US 12,380,925 US 12,382,188 US 12,387,403 US 12,393,613 US 12,401,772 US 12,405,672 US 12,405,675 US 12,406,448 US 12,411,555 US 12,412,347 US 12,423,910 US 12,429,953 US 12,432,441 US 12,437,491 US 12,443,270 US 12,443,325 US 12,443,335 US 12,450,851 US 12,452,384 US 12,456,263 US 12,462,507 US 12,468,395 US 12,468,396 US 12,468,439 US 12,469,090 US 12,469,219 US 12,482,171 US 12,494,078 US 12,498,783 US 12,498,793 US 12,513,254 US 12,517,521 US 12,518,490 US 12,524,146 US 12,530,086 US 12,530,088 US 12,531,959 US 12,547,295 US 12,547,309 US 12,572,216 US 12,572,261 US 12,573,100 US 12,608,126 US 12,614,567 US 12,626,455 US 12,632,984 US 12,633,061 US 12,638,951 US 12,641,190 US 12,645,280 US 12,650,732 US 12,656,875 US 12,656,876 US 12,657,753 US 12,670,643 US 12,670,674 US 12,675,168 US 12,675,169 US 12,693,770 US 12,700,187 US 12,705,009 US 12,710,807 US 12,712,835