System and method for automatic evaluation of gait using single or multi-camera recordings
Systems and methods in accordance with many embodiments of the invention include a motion evaluation system that trains a model to evaluate motion (such as, but not limited to, gait) through images (or video) captured by a single image capture device. In certain embodiments, motion evaluation includes predicting clinically relevant variables from videos of patients walking from keypoint trajectories extracted from the captured images.
1 . A method for evaluating motion from a video, the method comprising:
identifying a set of one or more keypoint trajectories from each of a plurality of frames of a video;
providing the set of one or more keypoint trajectories as inputs to an artificial neural network,
wherein the artificial neural network is trained using multivariate time series of coordinates of a set of one or more keypoint trajectories extracted from at least one training video, wherein each of the at least one training video is annotated with a true motion evaluation score,
wherein the artificial neural network is trained to predict motion evaluation scores based on the multivariate time series of coordinates and the true motion evaluation score of the corresponding training video where the multivariate time series of coordinates are extracted from,
wherein the artificial neural network comprises a convolutional neural network having one or more convolutional layers, wherein each of the one or more convolutional layers is configured to:
receive as input, a multivariate time series of coordinates structured as a T×D matrix, where T represents a number of points in a time dimension of the multivariate time series of coordinates, and D represents a depth dimension of the multivariate time series of coordinates; and
apply one or more filters of a filter length F to the multivariate time series of coordinates, wherein each filter connects to a local region in the time dimension comprising F consecutive points in the time dimension, and wherein each filter extends through an entirety of the depth dimension such that each filter connects to all D features at each of the F consecutive points in the time dimension;
computing the motion evaluation score based on the set of one or more keypoint trajectories using the artificial neural network; and
providing an output based on the motion evaluation score.
2 . The method of claim 1 , wherein identifying the set of keypoint trajectories comprises:
identifying a set of keypoints within each of a plurality of frames of the video;
computing each keypoint trajectory of the set of keypoint trajectories based on positions of the keypoint in each frame of the plurality of frames.
3 . The method of claim 2 , wherein the set of keypoints comprises two-dimensional (2D) positions of joints and body keypoints of an individual captured in the video.
4 . The method of claim 3 , wherein identifying the set of keypoints comprises using an OpenPose process to identify the set of keypoints.
5 . The method of claim 1 , wherein identifying the set of keypoint trajectories further comprises computing additional features from the identified set of keypoints.
6 . The method of claim 1 , wherein the motion evaluation score is one of gait deviation index (GDI), walking speed, cadence, symmetry, gait variability, and stride length.
7 . The method of claim 1 , wherein providing the output comprises providing a treatment regimen for a patient, based on the motion evaluation score.
8 . The method of claim 1 , wherein providing the output comprises providing a diagnosis for a disease.
9 . The method of claim 8 , wherein the disease is one of Parkinson's disease, osteoarthritis, stroke, cerebral palsy, multiple sclerosis, and muscular dystrophy.
10 . The method of claim 1 , wherein providing the output comprises providing the output for an individual's progression based on a plurality of predicted motion evaluation scores over a period of time.
11 . The method of claim 1 , wherein providing the output comprises providing real-time feedback to a user to adjust the user's motions.
12 . The method of claim 1 , wherein, the method further comprising:
performing a physics-based simulation based on the set of keypoint trajectories; and
training a model based on the physics-based simulation, wherein predicting the motion evaluation score comprises using the trained model to predict the motion evaluation score, wherein the motion evaluation score comprises at least one of muscle activation, muscle fiber length, and joint loads.
13 . A non-transitory machine readable medium containing processor instructions for evaluating motion from a video, where execution of the instructions by a processor causes the processor to perform a process that comprises:
identifying a set of one or more keypoint trajectories from a plurality of frames of a video;
providing the set of one or more keypoint trajectories as inputs to an artificial neural network,
wherein the artificial neural network is trained using multivariate time series of coordinates of a set of one or more keypoint trajectories extracted from at least one training video, wherein each of the at least one training video is annotated with a true motion evaluation score,
wherein the artificial neural network is trained to predict motion evaluation scores based on the multivariate time series of coordinates and the true motion evaluation score of the corresponding training video where the multivariate time series of coordinates are extracted from,
wherein the artificial neural network comprises a convolutional neural network having one or more convolutional layers, wherein each of the one or more convolutional layers is configured to:
receive as input, the multivariate time series of coordinates structured as a T×D matrix, where T represents a number of points in a time dimension of the multivariate time series of coordinates, and D represents a depth dimension of the multivariate time series of coordinates; and
apply one or more filters of a filter length F to the multivariate time series of coordinates, wherein each filter connects to a local region in the time dimension comprising F consecutive points in the time dimension, and wherein each filter extends through an entirety of the depth dimension such that each filter connects to all D features at each of the F consecutive points in the time dimension;
computing the motion evaluation score based on the set of one or more keypoint trajectories using the artificial neural network; and
providing an output based on the motion evaluation score.
14 . The non-transitory machine readable medium of claim 13 , wherein identifying the set of keypoint trajectories comprises:
identifying a set of keypoints within each of a plurality of frames of the video;
computing each keypoint trajectory of the set of keypoint trajectories based on positions of the keypoint in each frame of the plurality of frames.
15 . The non-transitory machine readable medium of claim 13 , wherein identifying the set of keypoint trajectories further comprises computing additional features from the identified set of keypoints.
16 . The non-transitory machine readable medium of claim 13 , wherein the motion evaluation score is one of gait deviation index (GDI), walking speed, cadence, symmetry, gait variability, and stride length.
17 . The non-transitory machine readable medium of claim 13 , wherein providing the output comprises providing the output for an individual's progression based on a plurality of predicted motion evaluation scores over a period of time.
18 . The non-transitory machine readable medium of claim 13 , wherein, the non-transitory machine readable medium, wherein the process further comprises:
performing a physics-based simulation based on the set of keypoint trajectories; and
training a model based on the physics-based simulation, wherein predicting the motion evaluation score comprises using the trained model to predict the motion evaluation score, wherein the motion evaluation score comprises at least one of muscle activation, muscle fiber length, and joint loads.