Detection of kinetic events and mechanical variables from uncalibrated video
Systems and techniques are provided to identify, analyze, and evaluate key events and mechanical variables in videos of human motion related to an action, such as may be used in training for various sports and other activities. Information about the action is calculated based on analysis of the video such as via keypoint identification, pose identification and/or estimation, and related calculations, and provided automatically to the user to allow for improvement of the action.
1. A computer-implemented method comprising:
receiving an uncalibrated video showing performance of an action by a human subject;
extracting a plurality of two-dimensional (2D) poses of the human subject while performing the action based on an arrangement of keypoints on the human subject in one or more frames of the uncalibrated video, the keypoints comprising repeatably-identifiable points on the human subject, wherein the 2D poses comprise discrete arrangements of portions of the human subject relative to one another;
computing one or more mechanical variables of the human subject performing the action, the mechanical variables describing a physical arrangement of at least a part of the human subject performing the action; and
based upon the one or more mechanical variables, automatically providing information about performance of the action by the human subject, the information indicating at least one portion of the action that was performed sub-optimally by the human subject.
2. The method of claim 1 , wherein the one or more mechanical variables are determined from one or more key events in the uncalibrated video, each of the key events corresponding to a predefined portion of the action corresponding to a specific temporal event occurring during performance of the action by the human subject.
3. The method of claim 2 , wherein the one or more key events are determined based upon the 2D poses.
4. The method of claim 1 , wherein the-plurality of 2D poses are selected from library of predefined two-dimensional (2D) signatures for the action based on the arrangement of keypoints.
5. The method of claim 4 , wherein the library of predefined two-dimensional (2D) signatures are generated prior to receiving the uncalibrated video.
6. The method of claim 4 , wherein each 2D signature in the library corresponds to a 2D projection of the action as seen from a corresponding point of view.
7. The method of claim 4 , wherein the 2D signatures are based upon a plurality of three-dimensional (3D) motion patterns previously obtained from 3D seed data.
8. The method of claim 7 , wherein the 3D seed data comprises motion capture data.
9. The method of claim 7 , wherein the 3D seed data comprises simulated motion data.
10. The method of claim 1 , wherein the information further comprises an identification of one or more exercises, drills, or activities to perform to cause an improvement in the portion of the action that was performed sub-optimally by the human subject.
11. The method of claim 1 , wherein the action is a pitch by a baseball pitcher.
12. The method of claim 11 , wherein the key events comprise one or more selected from the group consisting of: a front foot lift, a max leg lift, a foot strike, a max hip and shoulder separation, shoulders squared up to target, and ball release.
13. The method of claim 1 , wherein the action is a golf swing.
14. The method of claim 13 , wherein the key events comprise one or more selected from the group consisting of: a peak of the golf swing between a backswing and a downswing, a moment of ball impact, an initial follow-through, and a final position.
15. The method of claim 1 , wherein the action is selected from a group consisting of: a boxing punch, a basketball freethrow, a hockey slapshot, and a tennis swing.
16. A system comprising:
a processor configured to:
receive an uncalibrated video showing performance of an action by a human subject;
extract a plurality of two-dimensional (2D) poses of the human subject while performing the action based on an arrangement of keypoints on the human subject in one or more frames of the uncalibrated video, the keypoints comprising repeatably-identifiable points on the human subject, wherein the 2D poses comprise discrete arrangements of portions of the human subject relative to one another;
compute one or more mechanical variables of the human subject performing the action, the mechanical variables describing a physical arrangement of at least a part of the human subject performing the action; and
based upon the one or more mechanical variables, automatically provide information about performance of the action by the human subject, the information indicating at least one portion of the action that was performed sub-optimally by the human subject.
17. A non-transitory computer-readable medium storing a plurality of instructions which, when executed by a processor, cause the processor to:
receive an uncalibrated video showing performance of an action by a human subject;
extract a plurality of two-dimensional (2D) poses of the human subject while performing the action based on an arrangement of keypoints on the human subject in one or more frames of the uncalibrated video, the keypoints comprising repeatably-identifiable points on the human subject, wherein the 2D poses comprise discrete arrangements of portions of the human subject relative to one another;
compute one or more mechanical variables of the human subject performing the action, the mechanical variables describing a physical arrangement of at least a part of the human subject performing the action; and
based upon the one or more mechanical variables, automatically provide information about performance of the action by the human subject, the information indicating at least one portion of the action that was performed sub-optimally by the human subject.