Hand-eye, body part motion recognition and chronologically aligned display of recognized body parts
Provided is a motion recognition apparatus capable of individually recognizing motions performed using each body part. A motion recognition apparatus ( 10 ) includes: an acquisition unit ( 11 ) configured to acquire time series information regarding motions of an operator; a motion recognition unit ( 12 ) configured to recognize, on the basis of motion information that defines individual motions and the time series information, a motion of the operator corresponding to any one of the individual motions for each body part of the operator and generate a motion recognition result including a start time and an end time of the one of the individual motions corresponding to the recognized motion of the operator for each body part of the operator; and a display unit ( 13 ) configured to display the motion recognition result of each body part in a chronologically aligned manner on the basis of the generated motion recognition result.
1. A motion recognition apparatus comprising:
a processor configured to:
acquire time series information regarding motions of an operator; and
recognize, on the basis of motion information that defines individual motions and the time series information, a motion of the operator corresponding to any one of the individual motions for each body part of the operator, and generate a motion recognition result including a start time and an end time of the individual motion corresponding to the recognized motion of the operator for each body part of the operator, wherein the motion information comprises a corresponding relationship between the individual motions and motion elements in individual processes, the individual processes comprises an insertion process, a case fitting process, a soldering process and a screw fastening process; and
a display configured to display the motion recognition result of each body part in a chronologically aligned manner on the basis of the generated motion recognition result,
the motion recognition apparatus further comprising:
a learned model configured to have, as inputs, teacher data, the motion information and the time series information and have, as an output, information indicating the motion of the operator corresponding to any one of the individual motions, the teacher data that defines a start time or an end time of an identified motion for each body part.
2. The motion recognition apparatus according to claim 1 , wherein the processor inputs the teacher data, the motion information and the time series information to the learned model and recognizes the motion of the operator corresponding to any one of the individual motions for each body part of the operator on the basis of information indicating the motion of the operator corresponding to any one of the individual motions output from the learned model.
3. The motion recognition apparatus according to claim 1 , wherein the processor generates the motion recognition result by calculating the start time and the end time of each motion of the operator recognized for each body part of the operator and associating the calculated start time and the end time with the individual motion.
4. The motion recognition apparatus according to claim 1 ,
wherein the time series information is a video output from an image sensor, and
the processor extracts a skeleton of the operator on the basis of the video and recognizes the motion of the operator corresponding to any one of the individual motions for each body part of the operator on the basis of movement of the extracted skeleton.
5. The motion recognition apparatus according to claim 1 ,
wherein the time series information is a video output from an image sensor, and
the processor extracts feature amounts of movement of the operator from a video image of the video and recognizes the motion of the operator corresponding to any one of the individual motions for each body part of the operator on the basis of the extracted feature amount.
6. The motion recognition apparatus according to claim 1 , wherein the body parts of the operator include the right hand, the left hand, and the eyes of the operator.
7. The motion recognition apparatus according to claim 2 , wherein the processor generates the motion recognition result by calculating the start time and the end time of each motion of the operator recognized for each body part of the operator and associating the calculated start time and the end time with the individual motion.
8. The motion recognition apparatus according to claim 2 ,
wherein the time series information is a video output from an image sensor, and
the processor extracts a skeleton of the operator on the basis of the video and recognizes the motion of the operator corresponding to any one of the individual motions for each body part of the operator on the basis of movement of the extracted skeleton.
9. The motion recognition apparatus according to claim 3 ,
wherein the time series information s a video output from an image sensor, and
the processor extracts a skeleton of the operator on the basis of the video and recognizes the motion of the operator corresponding to any one of the individual motions for each body part of the operator on the basis of movement of the extracted skeleton.
10. The motion recognition apparatus according to claim 2 ,
wherein the time series information is a video output from an image sensor, and
the processor extracts feature amounts of movement of the operator from a video image of the video and recognizes the motion of the operator corresponding to any one of the individual motions for each body part of the operator on the basis of the extracted feature amount.
11. The motion recognition apparatus according to claim 3 ,
wherein the time series information is a video output from an image sensor, and
the processor extracts feature amounts of movement of the operator from a video image of the video and recognizes the motion of the operator corresponding to any one of the individual motions for each body part of the operator on the basis of the extracted feature amount.
12. The motion recognition apparatus according to claim 2 , wherein the body parts of the operator include the right hand, the left hand, and the eyes of the operator.
13. The motion recognition apparatus according to claim 3 , wherein the body parts of the operator include the right hand, the left hand, and the eyes of the operator.
14. The motion recognition apparatus according to claim 4 , wherein the body parts of the operator include the right hand, the left hand, and the eyes of the operator.
15. The motion recognition apparatus according to claim 5 , wherein the body parts of the operator include the right hand, the left hand, and the eyes of the operator.
16. A motion recognition method comprising:
acquiring time series information regarding motions of an operator;
recognizing, on the basis of motion information that defines individual motions and the time series information, a motion of the operator corresponding to any one of the individual motions for each body part of the operator and generating a motion recognition result including a start time and an end time of the individual motion corresponding to the recognized motion of the operator for each body part of the operator, wherein the motion information comprises a corresponding relationship between the individual motions and motion elements in individual processes, the individual processes comprises an insertion process, a case fitting process, a soldering process and a screw fastening process; and
displaying the motion recognition result of each body part in a chronologically aligned manner on the basis of the generated motion recognition result,
wherein teacher data, the motion information and the time series information are had as inputs of a learned model, information indicating the motion of the operator corresponding to any one of the individual motions is had as an output of the learned model, the teacher data that defines a start time or an end time of an identified motion for each body part.
17. A non-transitory computer readable storage medium storing motion recognition program that causes a computer to:
acquire time series information regarding motions of an operator;
recognize, on the basis of motion information that defines individual motions and the time series information, a motion of the operator corresponding to any one of the individual motions for each body part of the operator, and generate a motion recognition result including a start time and an end time of the individual motion corresponding to the recognized motion of the operator for each body part of the operator, wherein the motion information comprises a corresponding relationship between the individual motions and motion elements in individual processes, the individual processes comprises an insertion process, a case fitting process, a soldering process and a screw fastening process; and
display the motion recognition result of each body part in a chronologically aligned manner on the basis of the generated motion recognition result,
the computer further comprising:
a learned model configured to have, as inputs, teacher data, the motion information and the time series information and have, as an output, information indicating the motion of the operator corresponding to any one of the individual motions, the teacher data that defines a start time or an end time of an identified motion for each body part.
18. A motion recognition system comprising:
a sensor; and
a motion recognition apparatus,
wherein the sensor includes a detection unit configured to detect motions of an operator and output time series information regarding the motions,
the motion recognition apparatus includes
a processor configured to:
acquire the time series information,
recognize, on the basis of motion information that defines individual motions and the time series information, a motion of the operator corresponding to any one of the individual motions for each body part of the operator, and generate a motion recognition result including a start time and an end time of the individual motion corresponding to the recognized motion of the operator for each body part of the operator, wherein the motion information comprises a corresponding relationship between the individual motions and motion elements in individual processes, the individual processes comprises an insertion process, a case fitting process, a soldering process and a screw fastening process, and
a display configured to display the motion recognition result of each body part in a chronologically aligned manner on the basis of the generated motion recognition result,
the motion recognition apparatus further comprising:
a learned model configured to have, as inputs, teacher data, the motion information and the time series information and have, as an output, information indicating the motion of the operator corresponding to any one of the individual motions, the teacher data that defines a start time or an end time of an identified motion for each body part.