Gesture recognition apparatus, head-mounted-type display apparatus, gesture recognition method, and non-transitory computer readable medium for recognizing gesture using captured image
A gesture recognition apparatus according to the present invention includes at least one memory and at least one processor which function as: a first detection unit configured to detect from a captured image a first portion making a gesture; a second detection unit configured to detect from the captured image a second portion making the gesture in the first portion detected by the first detection unit; and a recognition unit configured to recognize the gesture on a basis of motion of the first portion detected by the first detection unit and motion of the second portion detected by the second detection unit, wherein in a case where a detection result satisfying a predetermined condition is not obtained by the second detection unit, the recognition unit recognizes the gesture using a detection result satisfying the predetermined condition obtained in a past by the second detection unit.
1. A gesture recognition apparatus comprising
at least one memory and at least one processor which function as:
a first detection unit configured to detect from a captured image from a first portion;
a second detection unit configured to detect from the captured image from a second portion in the first portion detected by the first detection unit; and
a recognition unit configured to recognize a gesture on a basis of motion of the first portion detected by the first detection unit and motion of the second portion detected by the second detection unit, wherein
in a case where a second portion satisfying a predetermined condition is not detected from a captured image at a predetermined timing by the second detection unit, the recognition unit recognizes a gesture using a second portion satisfying the predetermined condition detected from a captured image at a timing before the predetermined timing by the second detection unit and a first portion detected from the captured image at the predetermined timing by the first detection unit.
2. The gesture recognition apparatus according to claim 1 , wherein
in a case where a movement amount of a second portion in the captured image at the predetermined timing is larger than a threshold, the recognition unit uses the second portion satisfying the predetermined condition detected from the captured image at the timing before the predetermined timing by the second detection unit.
3. The gesture recognition apparatus according to claim 1 , wherein
the second detection unit detects a plurality of second portions, and
in a case where intervals between a plurality of second portions in the captured image at the predetermined timing are longer than a threshold, the recognition unit uses a plurality of second portions satisfying the predetermined condition detected from the captured image at the timing before the predetermined timing by the second detection unit.
4. The gesture recognition apparatus according to claim 1 , wherein
the first detection unit detects a plurality of first portions, and
in a case where a ratio of a part of one first portion, which is hidden by overlapping of another first portion, to the one first portion at the predetermined timing is larger than a threshold, the recognition unit uses, as a second portion in the one first portion, the second portion satisfying the predetermined condition detected from the captured image at the timing before the predetermined timing by the second detection unit.
5. The gesture recognition apparatus according to claim 1 , wherein
the at least one memory and the at least one processor further function as a determination unit configured to determine reliability of a second portion detected by the second detection unit, and
in a case where reliability of a second portion at the predetermined timing is lower than a threshold, the recognition unit uses a second portion having reliability higher than the threshold detected from the captured image at the timing before the predetermined timing by the second detection unit.
6. The gesture recognition apparatus according to claim 1 , wherein
the second detection unit cuts out from the captured image a region of the first portion detected by the first detection unit and detects the second portion from an image of the cut-out region.
7. The gesture recognition apparatus according to claim 1 , wherein
the at least one memory and the at least one processor further function as a generation unit configured to generate a control signal corresponding to the gesture recognized by the recognition unit.
8. The gesture recognition apparatus according to claim 1 , wherein
in a case where a second portion satisfying the predetermined condition is not detected from the captured image at the predetermined timing by the second detection unit, the recognition unit uses a newest second portion satisfying the predetermined condition detected from the captured image at the timing before the predetermined timing by the second detection unit.
9. The gesture recognition apparatus according to claim 1 , wherein
in a case where a second portion satisfying the predetermined condition is not detected from the captured image at the predetermined timing by the second detection unit, if a second portion satisfying the predetermined condition is not detected before the predetermined timing by the second detection unit, the recognition unit does not recognize a gesture.
10. The gesture recognition apparatus according to claim 1 , wherein
in a case where a second portion satisfying the predetermined condition is not detected from the captured image at the predetermined timing by the second detection unit, if a second portion satisfying the predetermined condition is not detected during a predetermined period up to the predetermined timing by the second detection unit, the recognition unit does not recognize a gesture.
11. The gesture recognition apparatus according to claim 1 , wherein
in a case where a second portion satisfying the predetermined condition is not detected from the captured image at the predetermined timing by the second detection unit, if a movement amount of the first portion in the captured image at the predetermined timing is larger than a threshold, the recognition unit does not recognize a gesture.
12. The gesture recognition apparatus according to claim 1 , wherein
the first portion is a hand, and the second portion is a joint.
13. A head-mounted-type display apparatus comprising
at least one memory and at least one processor which function as a display control unit configured to control display on a basis of a recognition result of the gesture recognition apparatus according to claim 1 .
14. The gesture recognition apparatus according to claim 1 , wherein
the at least one memory and the at least one processor further function as a recording unit configured to record, on a storage, a detection result of a second portion detected by the second detection unit, and
the recognition unit obtains, from the storage, a detection result of the second portion satisfying the predetermined condition detected from the captured image at the timing before the predetermined timing.
15. The gesture recognition apparatus according to claim 1 , wherein
in a case where a first portion is not detected by the first detection unit, the recognition unit does not recognize a gesture.
16. The gesture recognition apparatus according to claim 1 , wherein
in a case where a second portion satisfying the predetermined condition is not detected from the captured image at the predetermined timing by the second detection unit, the recognition unit uses, as a second portion in the first portion detected from the captured image at the predetermined timing, the second portion satisfying the predetermined condition detected from the captured image at the timing before the predetermined timing by the second detection unit.
17. The gesture recognition apparatus according to claim 1 , wherein
the first detection unit detects a rectangle region surrounding the first portion from the captured image, and
the second detection unit detects a spot corresponding to the second portion from the captured image.
18. The gesture recognition apparatus according to claim 1 , wherein
the first portion includes a plurality of second portions.
19. The gesture recognition apparatus according to claim 1 , wherein
the motion of the first portion is different from the motion of the second portion.
20. The gesture recognition apparatus according to claim 1 , wherein
the recognition unit detects the motion of the first portion on a basis of first portions detected from a plurality of captured images, and detects the motion of the second portion on a basis of second portions detected from the plurality of captured images.
21. A gesture recognition method comprising:
a first detection step of detecting from a captured image from a first portion;
a second detection step of detecting from the captured image from a second portion in the first portion detected in the first detection step; and
a recognition step of recognizing a gesture on a basis of motion of the first portion detected in the first detection step and motion of the second portion detected in the second detection step, wherein
in a case where a second portion satisfying a predetermined condition is not detected from a captured image at a predetermined timing in the second detection step, in the recognition step, a gesture is recognized using a second portion satisfying the predetermined condition detected from a captured image at a timing before the predetermined timing in a past second detection step and a first portion detected from the captured image at the predetermined timing in the first detection step.
22. A non-transitory computer readable medium that stores a program, wherein the program causes a computer to execute a gesture recognition method comprising:
a first detection step of detecting from a captured image from a first portion;
a second detection step of detecting from the captured image from a second portion in the first portion detected in the first detection step; and
a recognition step of recognizing a gesture on a basis of motion of the first portion detected in the first detection step and motion of the second portion detected in the second detection step, wherein
in a case where a second portion satisfying a predetermined condition is not detected from a captured image at a predetermined timing in the second detection step, in the recognition step, a gesture is recognized using a second portion satisfying the predetermined condition detected from a captured image at a timing before the predetermined timing in a past second detection step and a first portion detected from the captured image at the predetermined timing in the first detection step.