Skeleton recognition method, non-transitory computer-readable recording medium, and gymnastics scoring assist system
A skeleton recognition method includes extracting a plurality of first features presenting features of two-dimensional joint positions of a subject, based on two-dimensional input images that are input from a plurality of cameras that capture images of the subject generating, based on the first features, second feature group information containing a plurality of second features corresponding to a given number of joints of the subject, respectively sensing an abnormal second feature from the second feature group information and recognizing a 3D skeleton based on a result of integrating the second features that remain after removal of the abnormal second feature from the second feature group information, by using a processor.
1 . A skeleton recognition method comprising:
extracting a plurality of first features presenting features of two-dimensional joint positions of a subject, based on two-dimensional input images that are input from a plurality of cameras that capture images of the subject;
generating, based on the first features, second feature group information containing a plurality of second features corresponding to a given number of joints of the subject, respectively;
sensing an abnormal second feature from the second feature group information; and
recognizing a 3D skeleton based on a result of integrating the second features that remain after removal of the abnormal second feature from the second feature group information, by using a processor,
wherein the second feature is coordinates and heatmap information in which likelihood of presence of a given joint is associated with the coordinates, and
the sensing senses an abnormal feature based on a difference between the heatmap information and information of Gaussian distribution of likelihood that is specified previously.
2 . The skeleton recognition method according to claim 1 , wherein the generating generates a plurality of sets of second feature group information in time series, and
the sensing senses an abnormal second feature based on a first vector in which a given pair of joints that are specified based on previous second feature group information serves as a start point and an end point and a second vector in which a given pair of joints that are specified based on current second feature group information serves as a start point and an end point.
3 . The skeleton recognition method according to claim 2 , wherein the sensing senses an abnormal second feature based on a relationship between an area that is specified from a given joint based on the second feature group information and positions of joints other than the given joint.
4 . The skeleton recognition method according to claim 1 , wherein the sensing calculates a plurality of epipolar lines using camera positions as viewpoints based on the heatmap information and sensing an abnormal second feature based on a distance between an intersection of the epipolar lines and a position of a joint.
5 . A non-transitory computer-readable recording medium having stored therein a skeleton recognition program that causes a computer to execute a process comprising:
extracting a plurality of first features presenting features of two-dimensional joint positions of a subject, based on two-dimensional input images that are input from a plurality of cameras that capture images of a subject;
generating, based on the first features, second feature group information containing a plurality of second features corresponding to a given number of joints of the subject, respectively;
sensing an abnormal second feature from the second feature group information; and
recognizing a 3D skeleton based on a result of integrating the second features that remain after removal of the abnormal second feature from the second feature group information,
wherein the second feature is coordinates and heatmap information in which likelihood of presence of a given joint is associated with the coordinates, and
the sensing senses an abnormal feature based on a difference between the heatmap information and information of Gaussian distribution of likelihood that is specified previously.
6 . The non-transitory computer-readable recording medium according to claim 5 , wherein the generating generates a plurality of sets of second feature group information in time series, and
the sensing senses an abnormal second feature based on a first vector in which a given pair of joints that are specified based on previous second feature group information serves as a start point and an end point and a second vector in which a given pair of joints that are specified based on current second feature group information serves as a start point and an end point.
7 . The non-transitory computer-readable recording medium according to claim 6 , wherein the sensing senses an abnormal second feature based on a relationship between an area that is specified from a given joint based on the second feature group information and positions of joints other than the given joint.
8 . The non-transitory computer-readable recording medium according to claim 5 , wherein the sensing calculates a plurality of epipolar lines using camera positions as viewpoints based on the heatmap information and sensing an abnormal second feature based on a distance between an intersection of the epipolar lines and a position of a joint.
9 . A gymnastics scoring assist system including a plurality of cameras that capture images of a subject and a skeleton recognition apparatus comprising:
a processor configured to:
acquire two-dimensional input images that are input from the cameras;
extract a plurality of first features presenting features of two-dimensional joint positions of the subject based on the input images
generate, based on the first features, second feature group information containing a plurality of second features corresponding to a given number of joints of the subject, respectively;
sense an abnormal second feature from the second feature group information; and
recognize a 3D skeleton based on a result of synthesizing second features that remain after removal of the abnormal second feature from the second feature group information,
wherein the second feature is coordinates and heatmap information in which likelihood of presence of a given joint is associated with the coordinates and the processor is further configured to sense an abnormal feature based on a difference between the heatmap information and information of Gaussian distribution of likelihood that is specified previously.
10 . The gymnastics scoring assist system according to claim 9 , wherein the processor is further configured to generate a plurality of sets of second feature group information in time series and sense an abnormal second feature based on a first vector in which a given pair of joints that are specified based on previous second feature group information serves as a start point and an end point and a second vector in which a given pair of joints that are specified based on current second feature group information serves as a start point and an end point.
11 . The gymnastics scoring assist system according to claim 10 , wherein the processor is further configured to sense an abnormal second feature based on a relationship between an area that is specified from a given joint based on the second feature group information and positions of joints other than the given joint.
12 . The gymnastics scoring assist system according to claim 9 , wherein the processor is further configured to calculate a plurality of epipolar lines using camera positions as viewpoints based on the heatmap information and sense an abnormal second feature based on a distance between an intersection of the epipolar lines and a position of a joint.