Granted Patent B2

US 12,243,247 · App. 17/314,082 · Granted Mar 4, 2025

Inventor: Paul Oh (Seongnam-si, KR)

Assignee: Samsung Electronics Co., Ltd.

G06T7/251G06T7/75G06V40/28G06T2200/04G06T2207/30196

Abstract

Claims (82)

1. A method, the method comprising:

performing respective rotation conversions of initial positions of hand joints based on a set three-dimensional (3D) coordinate system, wherein the initial positions are obtained from an image of a hand and reflect a current pose of the hand;

estimating angle information, of the current pose, of finger joints resulting from the respective rotation conversions of the initial positions of the hand joints, with at least some of the respective rotation conversions corresponding to different rotations;

generating a kinematic model of the hand joints based on the initial positions of the hand joints and the angle information of the finger joints; and

tracking the hand joints by generating at least one of updated positions or angles of the hand joints in the current pose based on the initial positions of the hand joints and the kinematic model,

wherein the estimating of the angle information further includes performing the estimating of the angle information of the finger based on a selected, based on the determination of whether the finger joint is bent, one among a consideration of a geometry among multiple finger joints in the image and a consideration of a calculated finger joint angle in a previous image.

2. The method of claim 1 , wherein the performing of the respective rotation conversions comprises

performing the respective rotation conversions of the initial positions to a same plane for each of the finger joints based on a set 3D local coordinate system as the set 3D coordinate system.

3. The method of claim 2 , further comprising:

setting the 3D local coordinate system based on pose information of a palm connected to the finger joints, including:

estimating the pose information of the palm from position information of joints of the palm; and

generating 3D local coordinate axes, of the 3D local coordinate system, having a point in the palm as the origin based on the pose information of the palm.

4. The method of claim 2 , further comprising setting the 3D local coordinate system based on pose information of a palm connected to the finger joints, and

wherein the performing of the respective rotation conversions for each of the finger joints further comprises:

determining whether the finger joints are bent based on the 3D local coordinate system; and

adaptively calculating rotation conversion information for each of the finger joints based on a result of the determining of whether the finger joints are bent.

5. The method of claim 4 , wherein the performing of the respective rotation conversions includes performing the respective rotation conversions for all the hand joints according to a rotation matrix for rotating all the hand joints based on the 3D local coordinate system, and

wherein the determining of whether the finger joints are bent comprises:

calculating a basis of the plane on which each finger is located, in response to all the hand joints being on the plane due to the respective rotation conversions;

estimating yaw angles of the finger joints according to a bending estimation parameter based on the basis; and

determining whether the finger joints are bent based on a comparison of the estimated yaw angles to a threshold.

6. The method of claim 4 , wherein the adaptively calculating of the rotation conversion information comprises:

estimating first roll angles of the finger joints based on a geometry of the finger joints in the image, in response to a result of the determining being that the finger joints are bent; and

calculating pitch angles and yaw angles of the finger joints based on the first roll angles.

7. The method of claim 4 , wherein the adaptively calculating of the rotation conversion information comprises:

estimating second roll angles of at least some of the finger joints based on roll angles of the at least some finger joints in a previous image and initial roll angles that are set for the at least some finger joints, in response to a result of the determining being that the finger joints are stretched; and

calculating pitch angles and yaw angles of the at least some finger joints based on the second roll angles.

8. The method of claim 7 , wherein the estimating of the second roll angles comprises estimating the second roll angles of the some of the finger joints further based on angles of other fingers in the image.

9. The method of claim 1 , wherein the tracking comprises:

updating at least one of the initial positions of the finger joints or the angle information of the finger joints, respectively as the generated updated positions or angles, by an extended Kalman filter that is based on the kinematic model.

10. The method of claim 9 , wherein the updating of the at least one of the initial positions of the finger joints or the angle information of the finger joints comprises:

setting, as a state of the image, angles of the hand joints and a position and velocity of a wrist that are obtained by the extended Kalman filter;

predicting positions and angles of the hand joints in a future image from the state of the image; and

generating the updated positions and angles of the hand joints based on the predicted positions and angles.

11. The method of claim 1 , wherein the initial positions and angle information of the hand joints are dependent on a positional relationship between hand joints that are connected to each other.

12. The method of claim 1 , wherein some of the initial positions and angle information of the hand joints are determined based on a position of a hand joint, adjacent to a wrist, among hand joints connected to each other and an angle at which the hand joint is bent relative to a position of the wrist.

13. The method of claim 1 , wherein the image includes image frames, and

the method further comprises calculating the initial positions from the image frames.

14. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the processor to perform the method of claim 1 .

15. An apparatus, the apparatus comprising:

one or more processors configured to:

perform respective rotation conversions of initial positions of hand joints based on a set three-dimensional (3D) coordinate system, wherein the initial positions are obtained from an image of a hand and reflect a current pose of the hand;

estimate angle information, of the current pose, of finger joints resulting from the respective rotation conversions of the initial positions of the hand joints, with at least some of the respective rotation conversions corresponding to different rotations;

generate a kinematic model of the hand joints based on the initial positions of the hand joints and the angle information of the finger joints; and

track the hand joints by generating at least one of updated positions or angles of the hand joints in the current pose based on the initial positions of the hand joints and the kinematic model,

wherein the estimating of the angle information further includes performing the estimating of the angle information of the finger based on a selected, based on the determination of whether the finger joint is bent, one among a consideration of a geometry among multiple finger joints in the image and a consideration of a calculated finger joint angle in a previous image.

16. The apparatus of claim 15 , wherein, for the performing of the respective rotation conversion, the one or more processors are configured to perform the respective rotation conversions of the initial positions to a same plane based on a 3D local coordinate system as the set 3D coordinate system.

17. The apparatus of claim 16 , wherein the one or more processors are further configured to set the 3D local coordinate system based on pose information of a palm connected to the finger joints, including an estimation of the pose information of the palm from position information of joints of the palm, and a generation of 3D local coordinate axes, of the 3D local coordinate system, having a point in the palm as the origin based on the pose information of the palm.

18. The apparatus of claim 16 ,

wherein the one or more processors are further configured to set the 3D local coordinate system based on pose information of a palm connected to the finger joints, and

wherein, for the respective rotation conversions for each of the finger joints, the one or more processors are further configured to determine whether the finger joints are bent based on the 3D local coordinate system, and to adaptively calculate rotation conversion information for each of the finger joints based on a result of the determining of whether the finger joints are bent.

19. The apparatus of claim 18 ,

wherein, for the respective rotation conversions for each of the finger joints, the one or more processors are configured to performing the respective rotation conversions for all the hand joints according to a rotation matrix for rotating all the hand joints based on the 3D local coordinate system; and

wherein, for the determining of whether the finger joints are bent, the one or more processors are configured to:

calculate a basis of the plane on which each finger is located, in response to all the hand joints being on the plane due to the respective rotation conversions;

estimate yaw angles of the finger joints according to a bending estimation parameter based on the basis; and

determine whether the finger joints are bent based on a comparison of the estimated yaw angles to a threshold.

20. The apparatus of claim 15 , wherein, for the generating of the at least one of the updated positions or angles, the one or more processors are configured to update at least one of the initial positions of the finger joints or the angle information of the finger joints, respectively as the updated position or angles, by an extended Kalman filter that is based on the kinematic model.

21. An apparatus for tracking hand joints, the apparatus comprising:

one or more processors configured to:

estimate three-dimensional (3D) position information of hand joints from an image of a hand;

estimate angle information of the hand joints based on the 3D position information of the hand joints;

generate a kinematic model of the hand joints based on the 3D position information and the angle information of the hand joints;

update at least one of the 3D position information or the angle information of the hand joints by an extended Kalman filter based on the kinematic model; and

track the hand joints based on the updated position information or the updated angle information,

wherein the 3D local coordinate system is set based on pose information of a palm connected to finger joints of the hand, and

wherein the estimating of the angle information further includes:

a determination of whether the finger joints are bent based on the 3D local coordinate system; and

an estimation of first roll angles of the finger joints based on a geometry of the finger joints in the image, in response to a result of the determination being that the finger joints are bent.

22. The apparatus of claim 21 , wherein, for the estimation of the angle information, the one or more processors are further configured to:

estimate second roll angles of the finger joints based on roll angles of the finger joints in a previous image and initial roll angles that are set for the finger joints, in response to the result of the determination being that the finger joints are not bent; and

calculate pitch angles and yaw angles of the finger joints based on one of the first roll angles and the second roll angles.

23. The apparatus of claim 20 , further comprising a communication interface configured to transmit a result of the tracking of the hand joints to outside the apparatus.

24. The apparatus of claim 20 , further comprising an image sensor that captures the image.

25. The apparatus of claim 20 , further comprising a memory, wherein the one or more processors are further configured to store, in the memory, the angle information of the hand joints, the 3D position information of the hand joints, the updated 3D position information, and the updated angle information of the hand joints.

26. The method of claim 1 , wherein the tracking comprises updating at least one of the initial positions of the finger joints or the angle information of the finger joints by predicting positions of the hand joints in a future image and generating the updated positions of the hand joints based on the predicted positions, and/or predicting angles of the hand joints in the future image and generating the updated angles of the hand joints based on the predicted angles.

27. A method, the method comprising:

performing respective rotation conversions of initial positions of hand joints based on a set three-dimensional (3D) coordinate system, wherein the initial positions are obtained from an image of a hand and reflect a current pose of the hand;

estimating angle information, of the current pose, of finger joints resulting from the respective rotation conversions of the initial positions of the hand joints, with at least some of the respective rotation conversions corresponding to different rotations;

generating a kinematic model of the hand joints based on the initial positions of the hand joints and the angle information of the finger joints; and

tracking the hand joints by generating at least one of updated positions or angles of the hand joints in the current pose based on the initial positions of the hand joints and the kinematic model,

wherein the estimating of the angle information further includes a first estimation of respective first angles of one or more of the finger joints based on a geometry of multiple finger joints in the image, and a second estimation of respective second angles of a different one or more of the finger joints based on consideration of calculated angles of finger joints in a previous image.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (20)