Quantitative, biomechanical-based analysis with outcomes and context
Systems and methods are disclosed for generating a 3D avatar using a biomechanical analysis of observed actions with a focus on representing actions through computer-generated 3D avatars. Physical quantities of biomechanical actions can be measured from the observations, and the system can analyze these values, compare them to target or optimal values, and use the observations and known biomechanical capabilities to generate 3D avatars.
1. A computer-implemented system configured for applying biomechanical analysis to a sequence of images of a user's movement during performance of a golf swing to generate computer-generated three-dimensional (3D) avatars of the user's golf swing based on user selected kinematic parameters and sequences of motion, the system comprising:
one or more hardware processors configured by machine-readable instructions to:
receive, from at least one image capture device, the images of the user's movement during the performance of the golf swing;
provide a watch list engine configured to enable a user to select via a user interface one or more golf-specific, kinematic parameters and sequences of motion during the golf swing;
provide a movement module configured to quantitatively analyze the images to measure and track the user's golf swing in 3D space across the images, including to measure and track the user selected, golf-specific, kinematic parameters and sequences of motion, wherein the movement module is configured to identify one or more objects other than the user in the images, wherein the one or more objects includes at least a portion of a golf club, track kinematic parameters of the one or more objects in the images, and generate context data based on the tracked kinematic parameters of the one or more objects, and wherein the movement module is configured to identify pre-shot routines, wherein the user swings but does not hit a ball;
provide a machine learning module configured to determine a user's standard pre-shot routine over time and determine when a pre-shot routine deviates from the standard pre-shot routine;
provide a 3D avatar engine configured to automatically generate a 3D avatar of the user's golf swing based on the quantitative measurement of the golf-specific, kinematic parameters and sequences of motion, where the 3D avatar engine is configured to generate the 3D avatar by tailoring component structures of the 3D avatar based on golf-specific criteria including body points that correspond with pose, movement, and/or alignment features related to golf; and
provide a display generator configured to: i) display the 3D avatar of the user's golf swing and animate the 3D avatar to cause movement of the 3D avatar based on the measured, quantified movement from the sequence of images; and ii) display the user selected kinematic parameters with the 3D avatar of the user's golf swing.
2. The computer-implemented system of claim 1 wherein the watch list engine is configured to enable creation of one or more watch lists comprising a list of measurable attributes of the golf swing for which the system is configured to monitor during analysis of the images.
3. The computer-implemented system of claim 1 wherein the watch list engine is configured to enable creation of one or more golf-specific watch lists.
4. The computer-implemented system of claim 1 wherein the watch list engine is configured to enable creation of one or more golf-specific watch lists and specification of a weighting of kinematic parameters and sequences for analysis of images of golf swings, wherein the kinematic parameters correspond with measurable attributes of the golf swings.
5. The computer-implemented system of claim 1 configured to display a graphical user interface with options to enable creation of one or more motion trackers comprising specific kinematic parameters and sequences of motions.
6. The computer-implemented system of claim 1 configured to display a graphical user interface with options to enable creation of one or more golf-specific motion trackers.
7. The computer-implemented system of claim 1 configured to display a graphical user interface with options to enable creation of one or more motion trackers for tracking the movement of golf-specific kinematic parameters and sequences of motions.
8. The computer-implemented system of claim 1 configured to display a graphical user interface with options to enable creation of one or more motion trackers comprising specific kinematic parameters and sequences of motions, the kinematic parameters comprising position, velocity, acceleration, and/or higher-order derivatives of the position with respect to time.
9. The computer-implemented system of claim 1 wherein the 3D avatar engine is configured to apply recorded 3D motion data from the user to an avatar.
10. The computer-implemented system of claim 1 wherein the 3D avatar engine is configured to model the user's body as an articulated structure including a set of human skeletal components connected via joints and a measured spatial relationship among the user's skeletal components and animate the model along the 3D space based on the measured movement of the user along the 3D space.
11. The computer-implemented system of claim 1 wherein the 3D avatar engine is configured to customize a model of the user's body based on the golf-specific, kinematic parameters and sequences of motion.
12. The computer-implemented system of claim 1 , wherein the processor is further configured to compare, for the user selected, kinematic parameters and sequences of motion, the measured movements in the performance of the golf swing by the user to a stored set of target measured movements of a reference performance of the golf swing.
13. The computer-implemented system of claim 1 , wherein the processor is further configured to compare, for the user selected, kinematic parameters and sequences of motion, the measured movements in the performance of the golf swing by the user to a stored set of target measured movements of a reference performance of the golf swing; and the 3D avatar engine is configured to simultaneously display an avatar of the user's golf swing and animate the 3D avatar to cause movement of the 3D avatar based on the measured, tracked movement from the sequence of images based on the user's golf swing during the performance of the golf swing and a reference avatar based on the reference performance of the golf swing.
14. The computer-implemented system of claim 1 , wherein the processor is further configured to compare the measured movements in the user's performance of the golf swing to a stored set of a measured movements of a second user's performance of the golf swing.
15. The computer-implemented system of claim 1 , wherein the processor is further configured to compare, for the user selected, kinematic parameters and sequences of motion, the measured movements of the user's performance of the golf swing by the user to a stored set of a measured movements, including the user selected, kinematic parameters and sequences of motion, of a second user's performance of the golf swing; and the 3D avatar engine is configured to simultaneously display an avatar based on the user's performance of the activity and an avatar based on the second user's performance of the golf swing.
16. The computer-implemented system of claim 1 , wherein the processor is further configured to compare the measured movements in a user's first performance of the golf swing to a stored set of target measured movements of the user's second performance of the golf swing.
17. The computer-implemented system of claim 1 , wherein the processor is further configured to compare, for the user selected, kinematic parameters and sequences of motion, the measured movements in a user's first performance of the golf swing by the user to a stored set of target measured movements, including the user selected, golf-specific kinematic parameters and sequences of motion, of the user's second performance of the golf swing; and the 3D avatar engine is configured to simultaneously display an avatar based on the user's first performance of the golf swing and an avatar based on the user's second performance of the golf swing.
18. The computer-implemented system of claim 1 wherein the display generator is configured to enable a user to playback the user's performances of the golf swing, including display of a video of the user's performance and the animated 3D avatar display of the user's performance; and further comprising user selectable display elements that enable the user to control a perspective from which the animated 3D avatar is displayed.
19. The computer-implemented system of claim 1 , wherein the movement module is configured to identify one or more objects other than the user in the images, track kinematic parameters of the one or more objects in the images, and generate context data based on the tracked kinematic parameters of the one or more objects.
20. The computer-implemented system of claim 1 , wherein the movement module is configured to identify one or more objects other than the user in the images, track and measure kinematic parameters of the one or more objects in the images and the display generator is configured to display the 3D avatar relative to the one or more objects based on the measured movements of the user in the images and the kinematic parameters of the one or more objects in the images.
21. The computer-implemented system of claim 1 , wherein the movement module is further configured to measure and quantify user-selected movements of the user via a motion tracker, wherein the motion tracker is configured to generate measurements of a predetermined sequence of motions.
22. The computer-implemented system of claim 1 , wherein the movement module is further configured to:
dynamically generate one or more watch lists comprising a list of measurable attributes of a performance for which the system is programmed to monitor in the images of the user's movement during performance of the golf swing; and
store the watch lists in a memory of the system.
23. The computer-implemented system of claim 1 , wherein the movement module is further configured to:
display a graphical user interface with options to for user selection of one or more components of the user's movement during performance of the golf swing; and
store the one or more components in a memory of the system as one or more motion trackers.
24. The computer-implemented system of claim 1 , wherein the movement module is further configured to track a continuous movement of the golf swing across a sequence of images.
25. The computer-implemented system of claim 1 , wherein the 3D avatar engine is further configured to generate the 3D avatar based on a user selected sequence of measurements.
26. The computer-implemented system of claim 1 wherein the 3D avatar engine is further configured to:
i) model the user's body as an articulated structure, including a set of human skeletal components connected via joints and a measured spatial relationship among the user's skeletal components;
ii) customize the model of the user's body based on the performance of the golf swing being golf-specific, the customization including a determination of the human skeletal components and/or joints to include in the model based on at least the golf swing; and
iii) animate the model along three-dimensional 3D space based on the measured movement of the user along three-dimensional the 3D space.
27. The computer-implemented system of claim 1 wherein the at least one image capture device comprises a single camera and the 3D avatar engine is further configured to automatically generate the 3D avatar of the user from a single-viewpoint recording by the single camera.
28. The computer-implemented system of claim 1 , further comprising:
a watch list engine configured to generate a dynamic watch list;
a user interface configured to present options for a user to make selections for the dynamic watch lists by selecting one or more components, including one or more body parts or context objects; and
one or more motion trackers configured to determine the kinematic parameters of the one or more selected components to assess performance of the golf swing.
29. The computer-implemented system of claim 1 , wherein the 3D avatar engine is further configured to automatically generate the 3D avatar of the user by conversion of a 2D pose estimation from an image into the 3D space.
30. The computer-implemented system of claim 1 , wherein the 3D avatar engine is further configured to:
automatically generate the 3D avatar of the user from the 2D image by modeling the user's body as an articulated structure, including a predetermined set of human skeletal components connected via joints and a measured spatial relationship among the user's skeletal components; and
detecting the joints through 3D pose estimation.
31. The computer-implemented system of claim 1 , further comprising:
a 3D avatar engine configured to apply recorded 3D motion data from the user to the 3D avatar for: 1) playback of the user's own golf swing to enable the user to watch their own golf swing from any angle; and ii) apply a second user's recorded kinematic parameter sequence to a second avatar, to enable a golf coach, whose swing is recorded to model performances for the user.