Classification of musculoskeletal form using machine learning model
An exercise feedback system receives exercise data such as images or video captured by client devices of users performing exercises. The exercise feedback system may access a machine learning model trained using image of a population of users. The images used for training may be labeled, for example, as having proper or improper musculoskeletal form. The exercise feedback system may determine a metrics describing the musculoskeletal form of a user by applying the trained machine learning model to images of the user as input features. The exercise feedback system may generate feedback for a certain exercise using the metrics based on output predictions of the model. The feedback can be provided to a client device of the user or a physical therapist for presentation.
1. A method comprising:
receiving one or more images captured by a camera of a client device of a user, the one or more images indicating musculoskeletal form of the user while performing an exercise;
extracting a set of features from the one or more captured images, the extracted features describing the musculoskeletal form of the user;
accessing a machine learning model that predicts whether the user had proper musculoskeletal form while performing the exercise, the machine learning model trained using features describing a musculoskeletal form in a plurality of images of each of a plurality of other users performing the exercise with and without proper musculoskeletal form;
determining a metric describing the musculoskeletal form of the user by applying the machine learning model to the extracted set of features from the one or more captured images as input features;
determining a range of position of at least a portion of a human body performing the exercise based on images of a population of users performing the exercise, where the range of position defines a target correct musculoskeletal form;
generating feedback for the exercise using at least the metric, the feedback comprising a graphic of the range of position overlaid on an image or video of the user while performing an exercise, where the graphic comprises a boundary indicating that the musculoskeletal form of the user satisfies the target correct musculoskeletal form if within the boundary that represents the range of position; and
providing the feedback to the client device for presentation to the user.
2. The method of claim 1 , wherein determining the metric comprises:
determining a level of similarity between at least one of the input features and one or more reference features of the at least one of the plurality of images labeled as indicating proper musculoskeletal form, wherein the metric is determined using at least the level of similarity.
3. The method of claim 2 , further comprising:
determining a classification of the musculoskeletal form of the user as improper musculoskeletal form responsive to determining that the level of similarity is less than threshold similarity; and
transmitting a notification associated with the classification to another client device of a physical trainer of the user.
4. The method of claim 2 , wherein the at least one of the input features indicates a range of motion of the user while performing the exercise, and wherein determining the metric comprises:
determining that the user is fatigued responsive to determining that the range of motion is less than a threshold range.
5. The method of claim 2 , wherein the at least one of the input features indicates an amount of time that the user took to perform a repetition of the exercise, and wherein determining the metric comprises:
determining that the user is fatigued responsive to determining that the amount of time is greater than a threshold time.
6. The method of claim 1 , wherein determining the metric comprises:
determining a physical attribute of the user; and
determining a subset of the plurality of images indicating musculoskeletal form of the plurality of users associated with users having the physical attribute.
7. The method of claim 6 , wherein the physical attribute indicates a value or range of values representing a height or a weight of the user.
8. The method of claim 1 , further comprising:
determining, from the one or more images, an object nearby the user while the user is performing the exercise; and
determining a position of a portion of a body of the user relative to another position of the object, the metric determined further based on the position of the portion of the body.
9. The method of claim 1 , wherein generating the feedback for the exercise comprises:
determining at least one type of exercise equipment to recommend to the user.
10. A non-transitory computer-readable storage medium storing instructions for image processing, the instructions when executed by a processor causing the processor to:
receive one or more images captured by a camera of a client device of a user, the one or more images indicating musculoskeletal form of the user while performing an exercise;
extract a set of features from the one or more captured images, the extracted features describing the musculoskeletal form of the user;
access a machine learning model that predicts whether the user had proper musculoskeletal form while performing the exercise, the machine learning model trained using features describing a musculoskeletal form in a plurality of images of each of a plurality of other users performing the exercise with and without proper musculoskeletal form;
determine a metric describing the musculoskeletal form of the user by applying the machine learning model to the extracted set of features from the one or more captured images as input features;
determine a range of position of at least a portion of a human body performing the exercise based on images of a population of users performing the exercise, where the range of position defines a target correct musculoskeletal form;
generate feedback for the exercise using at least the metric, the feedback comprising a graphic of the range of position overlaid on an image or video of the user while performing an exercise, where the graphic comprises a boundary indicating that the musculoskeletal form of the user satisfies the target correct musculoskeletal form if within the boundary that represents the range of position; and
provide the feedback to the client device for presentation to the user.
11. The non-transitory computer-readable storage medium of claim 10 , wherein determining the metric comprises:
determining a level of similarity between at least one of the input features and one or more reference features of the at least one of the plurality of images labeled as indicating proper musculoskeletal form, wherein the metric is determined using at least the level of similarity.
12. The non-transitory computer-readable storage medium of claim 11 , the instructions when executed by the processor causing the processor to:
determine a classification of the musculoskeletal form of the user as improper musculoskeletal form responsive to determining that the level of similarity is less than threshold similarity; and
transmit a notification associated with the classification to another client device of a physical trainer of the user.
13. The non-transitory computer-readable storage medium of claim 11 , wherein the at least one of the input features indicates a range of motion of the user while performing the exercise, and wherein determining the metric comprises:
determining that the user is fatigued responsive to determining that the range of motion is less than a threshold range.
14. The non-transitory computer-readable storage medium of claim 11 , wherein the at least one of the input features indicates an amount of time that the user took to perform a repetition of the exercise, and wherein determining the metric comprises:
determining that the user is fatigued responsive to determining that the amount of time is greater than a threshold time.
15. The non-transitory computer-readable storage medium of claim 10 , wherein determining the metric comprises:
determining a physical attribute of the user; and
determining a subset of the plurality of images indicating musculoskeletal form of the plurality of users associated with users having the physical attribute.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the physical attribute indicates a value or range of values representing a height or a weight of the user.
17. The non-transitory computer-readable storage medium of claim 10 , the instructions when executed by the processor causing the processor to:
determine, from the one or more images, an object nearby the user while the user is performing the exercise; and
determine a position of a portion of a body of the user relative to another position of the object, the metric determined further based on the position of the portion of the body.
18. The non-transitory computer-readable storage medium of claim 10 , wherein generating the feedback for the exercise comprises:
determining at least one type of exercise equipment to recommend to the user.