Systems and methods for displaying augmented-reality objects
A computer-implemented method for displaying augmented-reality objects may include capturing, by an augmented-reality system, an image of a user's hand. The method may also include applying a machine learning model to the image to identify a set of key points and determine a gesture of the hand based on the key points. In addition, the method may include identifying a targeted area of the image containing a portion of a finger based on the key points and the gesture and, subsequently, detecting a pair of edge contours of the finger by performing a computer vision method on the targeted area. Furthermore, the method may include extrapolating a location to place an augmented-reality object on the finger. Finally, the method may include displaying, by the augmented-reality system, the augmented-reality object at the location on the finger by tracking the key points. Various other methods, systems, and computer-readable media are also disclosed.
1. A computer-implemented method comprising:
capturing, by at least one camera of an augmented-reality system, an image of a user's hand;
applying a machine learning model to the image of the user's hand to:
identify a set of key points of the user's hand; and
determine a gesture of the user's hand based on the set of key points;
identifying a targeted area of the image of the user's hand, wherein the targeted area:
contains a portion of a finger of the user's hand based on the set of key points and the gesture; and
lacks a location to place an augmented-reality object on the finger;
detecting a pair of edge contours of the finger by performing a computer vision method on the targeted area of the image, wherein the computer vision method comprises vertical edge detection to separately calculate each edge contour of the pair of edge contours;
extrapolating, from the detected pair of edge contours, the location to place the augmented-reality object on the finger by extrapolating a relative distance between the location and the targeted area of the image; and
displaying, by the augmented-reality system, the augmented-reality object at the location on the finger by tracking the set of key points of the user's hand.
2. The method of claim 1 , wherein the augmented-reality system comprises at least one of:
a mobile computing device;
the camera; or
a visual display.
3. The method of claim 1 , wherein the image of the user's hand comprises at least one of:
a still image; or
a video.
4. The method of claim 1 , wherein applying the machine learning model to identify the set of key points comprises training the machine learning model using a set of images of hands to create a base model.
5. The method of claim 4 , wherein applying the machine learning model to determine the gesture of the user's hand comprises:
rotating the base model to match the user's hand; and
matching the set of key points to the base model.
6. The method of claim 1 , wherein applying the machine learning model to the image of the user's hand comprises:
detecting that the gesture of the user's hand does not meet a predetermined threshold for an expected gesture; and
prompting, by the augmented-reality system, the user to position the user's hand in the expected gesture.
7. The method of claim 1 , wherein detecting the pair of edge contours of the finger comprises:
distinguishing between the finger and background constructs in the targeted area of the image; and
applying predetermined parameters to fit the pair of edge contours around the finger in the targeted area of the image.
8. The method of claim 1 , wherein extrapolating the location to place the augmented-reality object comprises:
extrapolating the relative distance of the location from the pair of edge contours in the targeted area of the image; and
extrapolating a width of the finger at the location.
9. The method of claim 1 , wherein displaying the augmented-reality object at the location comprises:
matching an orientation of the augmented-reality object to an orientation of the finger using the tracked set of key points; and
displaying the augmented-reality object, rotated to match the orientation of the finger, at the location on the finger.
10. The method of claim 9 , wherein the orientation of the augmented-reality object comprises:
a size of the augmented-reality object; and
an angle of the augmented-reality object.
11. The method of claim 1 , wherein displaying the augmented-reality object at the location comprises smoothing a rendering of the augmented-reality object to reduce jitter.
12. An augmented-reality system comprising:
a capturing module, stored in memory, that captures, by at least one camera, an image of a user's hand;
an application module, stored in memory, that applies a machine learning model to the image of the user's hand to:
identify a set of key points of the user's hand; and
determine a gesture of the user's hand based on the set of key points;
an identification module, stored in memory, that identifies a targeted area of the image of the user's hand, wherein the targeted area:
contains a portion of a finger of the user's hand based on the set of key points and the gesture; and
lacks a location to place an augmented-reality object on the finger;
a detection module, stored in memory, that detects a pair of edge contours of the finger by performing a computer vision method on the targeted area of the image, wherein the computer vision method comprises vertical edge detection to separately calculate each edge contour of the pair of edge contours;
an extrapolation module, stored in memory, that extrapolates, from the detected pair of edge contours, the location to place the augmented-reality object on the finger by extrapolating a relative distance between the location and the targeted area of the image;
a display module, stored in memory, that displays the augmented-reality object at the location on the finger by tracking the set of key points of the user's hand; and
at least one hardware processor that executes the capturing module, the application module, the identification module, the detection module, the extrapolation module, and the display module.
13. The augmented-reality system of claim 12 , wherein the image of the user's hand comprises at least one of:
a still image; or
a video.
14. The augmented-reality system of claim 12 , wherein the application module applies the machine learning model to identify the set of key points by training the machine learning model using a set of images of hands to create a base model.
15. The augmented-reality system of claim 14 , wherein the application module applies the machine learning model to determine the gesture of the user's hand by:
rotating the base model to match the user's hand; and
matching the set of key points to the base model.
16. The augmented-reality system of claim 12 , wherein the application module applies the machine learning model to the image of the user's hand by:
detecting that the gesture of the user's hand does not meet a predetermined threshold for an expected gesture; and
prompting, by the augmented-reality system, the user to position the user's hand in the expected gesture.
17. The augmented-reality system of claim 12 , wherein the detection module detects the pair of edge contours of the finger by:
distinguishing between the finger and background constructs in the targeted area of the image; and
applying predetermined parameters to fit the pair of edge contours around the finger in the targeted area of the image.
18. The augmented-reality system of claim 12 , wherein the extrapolation module extrapolates the location to place the augmented-reality object by:
extrapolating the relative distance of the location from the pair of edge contours in the targeted area of the image; and
extrapolating a width of the finger at the location.
19. The augmented-reality system of claim 12 , wherein the display module displays the augmented-reality object at the location by:
matching an orientation of the augmented-reality object to an orientation of the finger using the tracked set of key points; and
displaying the augmented-reality object, rotated to match the orientation of the finger, at the location on the finger.
20. A computer-readable medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to:
capture, by at least one camera of an augmented-reality system, an image of a user's hand;
apply a machine learning model to the image of the user's hand to:
identify a set of key points of the user's hand; and
determine a gesture of the user's hand based on the set of key points;
identify a targeted area of the image of the user's hand, wherein the targeted area:
contains a portion of a finger of the user's hand based on the set of key points and the gesture; and
lacks a location to place an augmented-reality object on the finger;
detect a pair of edge contours of the finger by performing a computer vision method on the targeted area of the image, wherein the computer vision method comprises vertical edge detection to separately calculate each edge contour of the pair of edge contours;
extrapolate, from the detected pair of edge contours, the location to place the augmented-reality object on the finger by extrapolating a relative distance between the location and the targeted area of the image; and
display, by the augmented-reality system, the augmented-reality object at the location on the finger by tracking the set of key points of the user's hand.