IP Library › Granted Patent US 12,444,060
Granted Patent B2
US 12,444,060 · App. 18/460,338 · Granted Oct 14, 2025

Scalable real-time hand tracking

Inventors: Valentin Bazarevsky (San Jose, CA); Fan Zhang (Sunnyvale, CA); Andrei Tkachenka (Mountain View, CA); Andrei Vakunov (Mountain View, CA); Matthias Grundmann (San Jose, CA)
Assignee: GOOGLE LLC
G06T7/251G06T7/75G06V40/28G06T2207/20081G06T2207/30196
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,444,060
App. No.
18/460,338
Filed
Sep 1, 2023
Granted
Oct 14, 2025
Kind
B2
Art Unit
2647
USPC
382/103
Abstract

Example aspects of the present disclosure are directed to computing systems and methods for hand tracking using a machine-learned system for palm detection and key-point localization of hand landmarks. In particular, example aspects of the present disclosure are directed to a multi-model hand tracking system that performs both palm detection and hand landmark detection. Given a sequence of image frames, for example, the hand tracking system can detect one or more palms depicted in each image frame. For each palm detected within an image frame, the machine-learned system can determine a plurality of hand landmark positions of a hand associated with the palm. The system can perform key-point localization to determine precise three-dimensional coordinates for the hand landmark positions. In this manner, the machine-learned system can accurately track a hand depicted in the sequence of images using the precise three-dimensional coordinates for the hand landmark positions.

Claims (58)

1. A computing system for hand tracking, the system comprising:

one or more processors; and

one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising:

obtaining a first image frame, wherein the first image frame is descriptive of a hand comprising a palm;

processing the first image frame with a machine-learned palm detection model to generate one or more oriented bounding boxes associated with a position of the palm, wherein the position of the palm is determined with the machine-learned palm detection model based on one or more features in the first image frame, wherein the one or more oriented bounding boxes are descriptive of an orientation of at least one of the palm or the hand;

processing the one or more oriented bounding boxes with a machine-learned hand landmark model to determine a first plurality of hand landmark positions within the first image frame based at least in part on the one or more oriented bounding boxes;

generating data indicative of a hand skeleton corresponding to a first palm detected in the first image frame based at least in part on three-dimensional coordinates corresponding to the first plurality of hand landmark positions within the first image frame;

determining a set of finger states associated with the hand skeleton based at least in part on an accumulated angle of joints associated with each finger of the hand skeleton; and

determining whether the first image frame is associated with one or more of a plurality of gestures based at least in part on mapping the set of finger states to a set of pre-defined gestures.

2. The system of claim 1 , wherein the operations further comprise:

rotating the first image frame based on an orientation of the one or more oriented bounding boxes; and

wherein the first plurality of hand landmark positions are determined based on the rotated first image frame.

3. The system of claim 1 , wherein the operations further comprise:

performing a plurality of transformations on the first image frame based on the one or more oriented bounding boxes to generate a transformed first image frame; and

wherein the first plurality of hand landmark positions are determined based on the transformed first image frame.

4. The system of claim 3 , wherein the plurality of transformations comprise at least two of: rotating, scaling, or cropping.

5. The system of claim 1 , wherein the one or more oriented bounding boxes are descriptive of the orientation and location of the palm, and wherein the operations further comprise:

generating one or more second oriented bounding boxes based on the one or more oriented bounding boxes, wherein the one or more second oriented bounding boxes are associated with a location and orientation of the hand; and

wherein the first plurality of hand landmark positions are determined based at least in part on the one or more second oriented bounding boxes.

6. The system of claim 1 , wherein the machine-learned hand landmark model is configured to perform key-point localization using a learned consistent internal hand pose representation.

7. The system of claim 1 , wherein the machine-learned hand landmark model is configured to perform key-point localization to generate three-dimensional coordinates corresponding to the first plurality of hand landmark positions within an image frame region by mapping the first plurality of hand landmark positions within the image frame region to the three-dimensional coordinates, wherein the three-dimensional coordinates are indicative of locations within a corresponding image frame.

8. The system of claim 1 , wherein the operations further comprise:

processing the first image frame with the machine-learned palm detection model to generate one or more second oriented bounding boxes associated with a position of the hand.

9. The system of claim 1 , wherein the machine-learned palm detection model comprises an encoder-decoder feature extractor configured to extract one or more features indicative of a context for each of the image frames input to the machine-learned palm detection model, wherein the one or more features indicative of a context for each image frame input to the machine-learned palm detection model is indicative of at least one of:

a presence of a hand;

a presence of an arm;

a presence of a body;

a presence of a face; or

a position of the hand.

10. The system of claim 1 , wherein processing the first image frame with the machine-learned palm detection model to generate the one or more oriented bounding boxes associated with the position of the palm comprises:

identifying one or more contextual features in the first image frame;

determining one or more rigid objects in the first image frame based on the one or more contextual features;

determining the position of the palm based on the one or more rigid objects; and

generating the one or more oriented bounding boxes based on the position of the palm.

11. A computer-implemented method for hand tracking, the method comprising:

obtaining, by a computing system comprising one or more processors, a first image frame, wherein the first image frame is descriptive of a hand comprising a palm;

processing, by the computing system, the first image frame with a machine-learned palm detection model to generate one or more oriented bounding boxes associated with a position of the palm, wherein the position of the palm is determined with the machine-learned palm detection model based on one or more features in the first image frame, wherein the one or more oriented bounding boxes are descriptive of an orientation of at least one of the palm or the hand;

processing, by the computing system, the one or more oriented bounding boxes with a machine-learned hand landmark model to determine a first plurality of hand landmark positions within the first image frame based at least in part on the one or more oriented bounding boxes;

generating, by the computing system, data indicative of a hand skeleton corresponding to a first palm detected in the first image frame based at least in part on three-dimensional coordinates corresponding to the first plurality of hand landmark positions within the first image frame;

determining, by the computing system, a set of finger states associated with the hand skeleton based at least in part on an accumulated angle of joints associated with each finger of the hand skeleton; and

determining, by the computing system, whether the first image frame is associated with one or more of a plurality of gestures based at least in part on mapping the set of finger states to a set of pre-defined gestures.

12. The method of claim 11 , wherein the one or more oriented bounding boxes are determined based at least in part on an estimation of rigid objects in the first image frame.

13. The method of claim 11 , wherein the one or more oriented bounding boxes are estimated based at least in part on one or more contextually aware features extracted from the first image frame.

14. The method of claim 11 , wherein the machine-learned hand landmark model determines the first plurality of hand landmark positions based on a place and orient of the palm as indicated by the one or more oriented bounding boxes.

15. The method of claim 11 , further comprising:

generating, by the computing system, a cropped image based on the one or more bounding boxes and the first image frame; and

inputting, by the computing system, the cropped image into the machine-learned hand landmark model, wherein the first plurality of hand landmark positions are determined based at least in part on the cropped image.

16. The method of claim 15 , wherein the cropped image comprises an image frame region scaled to a pre-defined size.

17. One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations comprising:

obtaining a first image frame, wherein the first image frame is descriptive of a hand comprising a palm;

processing the first image frame with a machine-learned palm detection model to generate one or more oriented bounding boxes associated with a position of the palm, wherein the position of the palm is determined with the machine-learned palm detection model based on one or more features in the first image frame, wherein the one or more oriented bounding boxes are descriptive of an orientation of at least one of the palm or the hand;

processing the one or more oriented bounding boxes with a machine-learned hand landmark model to determine a first plurality of hand landmark positions within the first image frame based at least in part on the one or more oriented bounding boxes;

generating data indicative of a hand skeleton corresponding to a first palm detected in the first image frame based at least in part on three-dimensional coordinates corresponding to the first plurality of hand landmark positions within the first image frame;

determining a set of finger states associated with the hand skeleton based at least in part on an accumulated angle of joints associated with each finger of the hand skeleton; and

determining whether the first image frame is associated with one or more of a plurality of gestures based at least in part on mapping the set of finger states to a set of pre-defined gestures.

18. The one or more non-transitory computer-readable media of claim 17 , wherein at least one of the machine-learned palm detection model or the machine-learned hand landmark model crop the first image frame based on the one or more oriented bounding boxes.

19. The one or more non-transitory computer-readable media of claim 17 , wherein the operations further comprise:

generating a rendering based at least in part on the first plurality of hand landmark positions.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 7, 2023
From: BAZAREVSKY, VALENTIN; ZHANG, FAN; TKACHENKA, ANDREI; VAKUNOV, ANDREI; GRUNDMANN, MATTHIAS
To: GOOGLE LLC
Reel/Frame 064834/0119 →
Continuity (3)
Continuation 17527463 · Nov 16, 2021
Continuation 16709128 · Dec 10, 2019
Related Publication 20230410329A1 · Dec 21, 2023
References Cited (42)
US 10013070B2 · Ahn et al. · 2018 [cited by applicant]
US 10579869B1 · Xiong · 2020 [cited by examiner]
US 20150071487A1 · Sheng · 2015 [cited by examiner]
US 20150177846A1 · Yin · 2015 [cited by examiner]
US 20170038846A1 · Minnen · 2017 [cited by examiner]
US 20180024641A1 · Mao · 2018 [cited by examiner]
US 20180088671A1 · Wang · 2018 [cited by examiner]
US 20190034714A1 · Barth · 2019 [cited by examiner]
US 20190278983A1 · Iqbal · 2019 [cited by examiner]
US 20200193638A1 · Taylor · 2020 [cited by examiner]
US 20200301513A1 · Mejia Cobo · 2020 [cited by examiner]
US 20210052249A1 · Gafner · 2021 [cited by examiner]
US 20210081055A1 · Sinha · 2021 [cited by examiner]
US 20210183097A1 · Georgakis · 2021 [cited by examiner]
CN 106469268 · 2017 [cited by applicant]
CN 109086725 · 2021 [cited by applicant]
CN 109086725B · 2021 [cited by examiner]
CN 112183424 · 2021 [cited by applicant]
CN 112183424A · 2021 [cited by examiner]
EP 3686772A1 · 2020 [cited by applicant]
Ziaolong Zhu a Two-stage Detector for Hand Detection in Ego-centric Videos Published on 2016 (Year: 2016). [cited by examiner]
Hand Posture Estimation from 2D Image Sequence by Hand Landmark Identification—2017 (Year: 2017). [cited by examiner]
Bazarevsky et al., “BlazeFace: Sub-Millisecond Neural Face Detection on Mobile GPUs”, arXiv:1907.05047y2, Jul. 14, 2019, retrieved Dec. 10, 2019, 4 pages. [cited by applicant]
Bazarevsky et al., “On-Device, Real-Time Hand Tracking with MediaPipe”, Aug. 19, 2019, https://ai.googleblog.com/2019/08/on-device-real-time-hand-tracking-with.html, retrieved on Dec. 10, 2019, 7 pages. [cited by applicant]
Common Objects in Context, http://cocodataset.org/#detection-eval, retrieved Dec. 10, 2019, 3 pages. [cited by applicant]
Github, “Hand Tracking (GPU)”, Aug. 16, 2019, https://github.com/google/mediapipe/blob/master/mediapipe/docs/hand_tracking_mobile_gpu.mdm retrieved on Dec. 10, 2019, 21 pages. [cited by applicant]
Github, “Mediapipe”, Jun. 16, 2019, https://github.com/google/mediapipe/blob/master/mediapipe/docs/examples.md, retrieved Dec. 10, 2019, 5 pages. [cited by applicant]
Github, “Mediapipe”, Jun. 29, 2019, https://github.com/google/mediapipe/, retrieved Dec. 10, 2019, 3 pages. [cited by applicant]
Gomez-Donoso et al., “Robust Hand Pose Regression Using Convolutional Neural Networks”, Robot 2017: Third Iberian Robotics Conference, vol. 693, 2018, pp. 591-602. [cited by applicant]
Kartynnik et al., “Real-Time Facial Surface Geometry from Monocular Video on Mobile GPU's”, https://drive.google.com/file/d/1mWpYDANuvP7bcgbqjkjbDKxuYFO7QoKB/view, Conference on Computer Vision and Pattern Recognition, … [cited by applicant]
Lin et al., “Feature Pyramid Networks for Object Detection”, arXiv:1612.03144v2, Apr. 19, 2017, retrieved Dec. 10, 2019, 10 pages. [cited by applicant]
Lin et al., “Focal Loss for Dense Object Detection”, arXiv:1708.02002v2, Feb. 7, 2018, retrieved Dec. 10, 2019, 10 pages. [cited by applicant]
Liu et al., “SSD: Single Shot MultiBox Detector”, arXiv:1512:02325v5, Dec. 29, 2016, retrieved Dec. 10, 2019, 17 pages. [cited by applicant]
Papandreou et al., “Towards Accurate Multi-Person Pose Estimation in the Wild”, arXiv:1701:01779v2, Apr. 14, 2017, 9 pages. [cited by applicant]
Patel, Giving Lens New Reading Capabilities in Google Go, Sep. 4, 2019, https://ai.googleblog.com/search/label/Augmented%20Reality, retrieved on Dec. 10, 2019, 19 pages. [cited by applicant]
Sinion et al., “Hand Keypoint Detection in Single Images Using Multiview Bootstrapping”, arXiv:1704.07809v1, Apr. 25, 2017, 9 pages. [cited by applicant]
Sunyoto et al., “Wrist Detection Based on a Minimum Bounding Box and Geometric Features”, https://doi.org/10.1016/j.jksuci.2018.05.005, Journal of King Saud University—Computer and Information Sciences, 2018, 8 pages. [cited by applicant]
Tensorflow, “TensorFlow Lite on GPU”, https://www.tensorflow.org/lite/performance/gpu_advanced, retrieved on Dec. 10, 2019, 11 pages. [cited by applicant]
Ungureanu et al., “Toward Unconstrained Palmprint Recognition on Consumer Devices: A Literature Review”, arXiv:2003,00737vl, Mar. 2, 2020, 16 pages. [cited by applicant]
Yan et al., “A Neural Network Model for Learning 3D Object Representations Through Haptic Exploration”, Frontiers in Neurorobotics, vol. 15, Article 639001, 2021, pp. 1-15. [cited by applicant]
Zhang et al., “MediaPipe Hands: On-device Real-time Hand Tracking”, arXiv:2006.10214v1, Jun. 18, 2020. 5 pages. [cited by applicant]
Zhu et al., “A Two-stage Detector for Hand Detection in Ego-centric Videos”, 2016 Institute of Electrical and Electronics Engineers Winter Conference on Applications of Computer Vision (WACV), 2016, pp. 1-8, 9 pages. [cited by applicant]
Cited By (1)
US 12,718,622