Adaptive hand tracking and gesture recognition using face-shoulder feature coordinate transforms
A method for recognizing hand gestures when the hand is held in front of the user's upper body. The method detects dominant features associated with the upper body, including features on the face (eyes and mouth etc.) and the shoulder extremes. Geometrical relations among these features are used to judge whether the face/head has turned to the side relative to the shoulder. If not, a face-shoulder coordinate system is established using these face-shoulder features, and the hand images are transformed into this new coordinate system. This in effect defines hand gestures relative to the user's face and shoulder, thus significantly reducing the dimensions of the feature space for hand gestures. This face-shoulder coordinate transform can be combined either with traditional appearance-based hand gesture recognition methods, or with a conventional or hybrid deep neural network for RGB or RGB-D images.
1. A method for hand tracking and gesture recognition implemented in a computer system, comprising:
receiving a current image from a camera system, the current image containing a face and shoulders of a human user;
from the current image, detecting face and shoulder features of the user;
establishing a current face-shoulder coordinate system based on the detected face and shoulder features;
transforming a previous hand region of interest, which is a region of interest containing a hand of the user and which has been generated from a previous image received from the camera system, from a previous face-shoulder coordinate system into the current face-shoulder coordinate system to generate a transformed previous hand region of interest;
extracting a current hand region of interest from the current image, wherein the current hand region of interest is defined in the current face-shoulder coordinate system; and
performing gesture recognition using the transformed previous hand region of interest and the current hand region of interest in the current face-shoulder coordinate system.
2. The method of claim 1 ,
wherein the detecting step includes detecting the user's eyes, mouth and shoulders and computing current left and right eye center positions, a current mouth center position, and current left and right shoulder curvature extreme positions of the user, and
wherein the establishing step includes establishing the current face-shoulder coordinate system based on the current left and right eye center positions, the current mouth center position, and the current left and right shoulder curvature extreme positions.
3. The method of claim 2 , wherein the establishing step includes:
defining an origin of the face-shoulder coordinate system to be a midpoint between the current left and right shoulder curvature extremes;
defining an X direction of the face-shoulder coordinate system to be a direction from the origin to the current mouth center; and
defining a Y direction of the face-shoulder coordinate system to be a direction from the origin to the current left or right shoulder curvature extreme; and
defining a Z direction of the face-shoulder coordinate system to be a direction orthogonal to a plane formed by the current left and right shoulder curvature extremes and the current mouth center.
4. The method of claim 2 , further comprising, after the detecting step and before the establishing step:
computing a current topology of the current left and right eye center positions, the current mouth center position, and the current left and right shoulder curvature extreme positions; and
evaluating the current topology to confirm that a plane formed by the user's face and shoulder feature is within a predetermined angular range relative to an optical axis of the camera system.
5. The method of claim 2 , further comprising, after the detecting step and before the establishing step:
computing a current topology of the current left and right eye center positions, the current mouth center position, and the current left and right shoulder curvature extreme positions; and
determining that the current topology is an affine transformation of an initial topology which has been computed from initial left and right eye center positions, an initial mouth center position, and initial left and right shoulder curvature extreme positions.
6. The method of claim 5 , wherein each of the current topology and the initial topology is defined by three ratios including:
r
LR
=
E
R
S
R
E
L
S
L
(
1
)
r
LRT
=
E
R
M
C
E
L
M
C
(
2
)
r
LRB
=
S
R
M
C
S
L
M
C
(
3
)
where EL and ER are the current or initial left and right eye centers, MC is the current or initial mouth center, and SL and SR are the current or initial shoulder curvature extremes, and
wherein the current topology is determined to be an affine transformation of the initial topology when the three ratios of the current topology are respectively equal to the three ratios of the initial topology within predefined thresholds.
7. The method of claim 1 , wherein the detecting step includes:
obtaining a previous face-shoulder region of interest, which is a region of interest in the previous image that contains the face and shoulders of the user;
tracking features in the previous face-shoulder region of interest to features in the current image;
extracting the current face-shoulder region of interest from the current image based on the tracking; and
detecting the user's eyes, mouth and shoulders in the current face-shoulder region of interest.
8. The method of claim 1 , wherein the establishing step includes:
obtaining a previous face-shoulder region of interest, which is a region of interest in the previous image that contains the face and shoulders of the user, and a previous face-shoulder coordinate system which has been established for the previous image;
extracting a current face-shoulder region of interest, which is a region of interest in the current image that contains the face and shoulders of the user;
establishing a mapping that maps the previous face-shoulder region of interest to the current face-shoulder region of interest based on face-shoulder features or local image features in the respective face-shoulder regions of interest, the mapping including a rotation and a translation; and
applying the mapping to the previous face-shoulder coordinate system to generate the current face-shoulder coordinate system.
9. The method of claim 1 , wherein the step of extracting the current hand region of interest includes:
performing hand tracking in the current face-shoulder coordinate system using the transformed previous hand region of interest; and
extracting the current hand region of interest from the current image based on the hand tracking.
10. The method of claim 1 , wherein the step of extracting the current hand region of interest includes:
defining a hand search window in the current image based on user statistics or a predefined window size;
detecting a hand in the hand search window; and
extracting the current hand region of interest from the current image based the detected hand.
11. A computer program product comprising a non-transitory computer readable storage medium having a computer readable program code embedded therein for controlling a computer, the computer readable program code configured to cause the computer to execute a process for hand tracking and gesture recognition, the process comprising:
receiving a current image from a camera system, the current image containing a face and shoulders of a human user;
from the current image, detecting face and shoulder features of the user;
establishing a current face-shoulder coordinate system based on the detected face and shoulder features;
transforming a previous hand region of interest, which is a region of interest containing a hand of the user and which has been generated from a previous image received from the camera system, from a previous face-shoulder coordinate system into the current face-shoulder coordinate system to generate a transformed previous hand region of interest;
extracting a current hand region of interest from the current image, wherein the current hand region of interest is defined in the current face-shoulder coordinate system; and
performing gesture recognition using the transformed previous hand region of interest and the current hand region of interest in the current face-shoulder coordinate system.
12. The computer program product of claim 11 ,
wherein the detecting step includes detecting the user's eyes, mouth and shoulders and computing current left and right eye center positions, a current mouth center position, and current left and right shoulder curvature extreme positions of the user, and
wherein the establishing step includes establishing the current face-shoulder coordinate system based on the current left and right eye center positions, the current mouth center position, and the current left and right shoulder curvature extreme positions.
13. The computer program product of claim 12 , wherein the establishing step includes:
defining an origin of the face-shoulder coordinate system to be a midpoint between the current left and right shoulder curvature extremes;
defining an X direction of the face-shoulder coordinate system to be a direction from the origin to the current mouth center; and
defining a Y direction of the face-shoulder coordinate system to be a direction from the origin to the current left or right shoulder curvature extreme; and
defining a Z direction of the face-shoulder coordinate system to be a direction orthogonal to a plane formed by the current left and right shoulder curvature extremes and the current mouth center.
14. The computer program product of claim 12 , wherein the process further comprises, after the detecting step and before the establishing step:
computing a current topology of the current left and right eye center positions, the current mouth center position, and the current left and right shoulder curvature extreme positions; and
evaluating the current topology to confirm that a plane formed by the user's face and shoulder feature is within a predetermined angular range relative to an optical axis of the camera system.
15. The computer program product of claim 12 , wherein the process further comprises, after the detecting step and before the establishing step:
computing a current topology of the current left and right eye center positions, the current mouth center position, and the current left and right shoulder curvature extreme positions; and
determining that the current topology is an affine transformation of an initial topology which has been computed from initial left and right eye center positions, an initial mouth center position, and initial left and right shoulder curvature extreme positions.
16. The computer program product of claim 15 , wherein each of the current topology and the initial topology is defined by three ratios including:
r
LR
=
E
R
S
R
E
L
S
L
(
1
)
r
LRT
=
E
R
M
C
E
L
M
C
(
2
)
r
LRB
=
S
R
M
C
S
L
M
C
(
3
)
where EL and ER are the current or initial left and right eye centers, MC is the current or initial mouth center, and SL and SR are the current or initial shoulder curvature extremes, and
wherein the current topology is determined to be an affine transformation of the initial topology when the three ratios of the current topology are respectively equal to the three ratios of the initial topology within predefined thresholds.
17. The computer program product of claim 11 , wherein the detecting step includes:
obtaining a previous face-shoulder region of interest, which is a region of interest in the previous image that contains the face and shoulders of the user;
tracking features in the previous face-shoulder region of interest to features in the current image;
extracting the current face-shoulder region of interest from the current image based on the tracking; and
detecting the user's eyes, mouth and shoulders in the current face-shoulder region of interest.
18. The computer program product of claim 11 , wherein the establishing step includes:
obtaining a previous face-shoulder region of interest, which is a region of interest in the previous image that contains the face and shoulders of the user, and a previous face-shoulder coordinate system which has been established for the previous image;
extracting a current face-shoulder region of interest, which is a region of interest in the current image that contains the face and shoulders of the user;
establishing a mapping that maps the previous face-shoulder region of interest to the current face-shoulder region of interest based on face-shoulder features or local image features in the respective face-shoulder regions of interest, the mapping including a rotation and a translation; and
applying the mapping to the previous face-shoulder coordinate system to generate the current face-shoulder coordinate system.
19. The computer program product of claim 11 , wherein the step of extracting the current hand region of interest includes:
performing hand tracking in the current face-shoulder coordinate system using the transformed previous hand region of interest; and
extracting the current hand region of interest from the current image based on the hand tracking.
20. The method of claim 11 , wherein the step of extracting the current hand region of interest includes:
defining a hand search window in the current image based on user statistics or a predefined window size;
detecting a hand in the hand search window; and
extracting the current hand region of interest from the current image based the detected hand.