Systems and methods for computer vision and machine-learning based form feedback
A plurality of motion sensors capable of capturing IR video, depth video, and RGB images. Periodically emitting IR pulses. Capturing IR video, depth video, and RGB images of a user. Generating a point cloud of the user. Generating a 3D model of the user including the point cloud of the user. Inputting the IR video, depth video, and RGB images into a machine learning model. Estimating a set of joints of the user in the 3D model. Tracking, based on the estimated set of joints of the user in the 3D model, the user's motion over a period of time. Determining a number of repetitions of an exercise performed by the user. Determining a feedback value. Calculating, based on the number of repetitions and the feedback value, a user exercise score. Providing, via a graphical user interface, the user exercise score and the feedback value to the user.
1. A system comprising:
a plurality of three-dimensional (3D) motion sensors, the plurality of 3D motion sensors including one or more infrared (IR) sensors and one or more red, green, and blue (RGB) sensors, the plurality of 3D motion sensors being capable of capturing IR video, depth video, and RGB images;
one or more hardware processors;
memory storing instructions that, when executed by the one or more hardware processors, cause the system to perform:
periodically emitting, by at least a portion of the plurality of 3D motion sensors, IR pulses;
capturing, using the plurality of 3D motion sensors and at least partially based on the periodically emitted IR pulses, IR video of a user and exercise equipment being used by the user through a plurality of movements of an exercise, depth video of the user and the exercise equipment being used by the user through the plurality of movements of the exercise, and RGB images of the user and the exercise equipment being used by the user through the plurality of movements of the exercise;
generating, at least partially based on the captured IR pulses, a point cloud of the user and the exercise equipment being used by the user through the plurality of movements of the exercise;
generating, in response to generating the point cloud of the user and the exercise equipment, a 3D model of the user, the 3D model including the point cloud of the user and the exercise equipment;
inputting the IR video of the user, the depth video of the user, and the RGB images of the user into a machine learning model;
estimating a set of joints of the user in the 3D model, the estimating using the point cloud and the machine learning model with the inputted IR video, the inputted depth video, and the inputted RGB images;
tracking, based on the estimated set of joints of the user in the 3D model, the user's motion over a period of time;
determining, based on the tracked user's motion over the period of time and one or more exercise models, a number of repetitions of the exercise performed by the user over the period of time;
determining, based on the tracked user's motion over the period of time and one or more form feedback models, a form feedback value from a set of form feedback values, the set of form feedback values including a satisfactory value and an unsatisfactory value;
calculating, based on the number of repetitions and the form feedback value, a user exercise score;
providing, via a graphical user interface, the user exercise score and the form feedback value to the user, thereby instructing the user to adjust their form during subsequent repetitions of the exercise.
2. The system of claim 1 , wherein the instructions, when executed by the one or more hardware processors, cause the system to perform:
identifying, using the one or more RGB sensors, one or more weights associated with the exercise equipment;
tracking position and orientation and motion of the exercise equipment over the period of time;
wherein the form feedback value is determined based the tracked user's motion over the period of time and the tracked position and orientation and motion of the exercise equipment over the period of time, and the one or more form feedback models; and
wherein the user exercise score is calculated based on the one or more weights associated with the exercise equipment and the number of repetitions and the form feedback value.
3. The system of claim 2 , wherein the exercise equipment is of a particular color, and the one or more weights associated with the exercise equipment are identified based on the particular color.
4. The system of claim 1 , wherein the machine learning model comprises a convolutional neural net machine learning model.
5. The system of claim 4 , wherein the instructions, when executed by the one or more hardware processors, cause the system to perform:
validating, using another machine learning model and a statistical model, the estimated set of joints of the user in the 3D model prior to tracking the user's motion over the period of time.
6. The system of claim 1 , wherein the point cloud comprises at least 80,000 points, and the estimated set of joints comprises 25 points.
7. The system of claim 1 , wherein the graphical user interface comprises a graphical user interface presented on a screen display of a free-standing A-frame exercise equipment cabinet.
8. The system of claim 2 , wherein the instructions, when executed by the one or more hardware processors, cause the system to perform:
ranking the user, based on the user exercise score, relative to a plurality of other users, wherein each of the other users have a corresponding user exercise score;
providing, via the graphical user interface, a leaderboard including a user list, the user list based on the ranking of the user relative to the other users.
9. The system of claim 1 , wherein the instructions, when executed by the one or more hardware processors, cause the system to perform:
determining, based on the tracked user's motion over the period of time and one or more other machine learning models, a severity score associated with the exercise performed by the user over the period of time, the severity score indicating a degree of error in some or all of the movements of the exercise performed by the user;
notifying, based on the severity score and a dynamic severity threshold value, the user that the user has made an error.
10. They system of claim 1 , wherein the instructions, when executed by the one or more processors, cause the system to perform:
reducing, in response to determining the form feedback value and based on the form feedback value, the number of determined repetitions performed by the user over the period of time.
11. A method implemented by a computing system including a plurality of three-dimensional (3D) motion sensors, the plurality of 3D motion sensors including one or more infrared (IR) sensors and one or more red, green, and blue (RGB) sensors, the plurality of 3D motion sensors being capable of capturing IR video, depth video, and RGB images, and one or more processors and storage media storing machine-readable instructions, wherein the method is performed using the one or more processors, the method comprising:
periodically emitting, by at least a portion of the plurality of 3D motion sensors, IR pulses;
capturing, using the plurality of 3D motion sensors and at least partially based on the periodically emitted IR pulses, IR video of a user and exercise equipment being used by the user through a plurality of movements of an exercise, depth video of the user and the exercise equipment being used by the user through the plurality of movements of the exercise, and RGB images of the user and the exercise equipment being used by the user through the plurality of movements of the exercise;
generating, at least partially based on the captured IR pulses, a point cloud of the user and the exercise equipment being used by the user through the plurality of movements of the exercise;
generating, in response to generating the point cloud of the user and the exercise equipment, a 3D model of the user, the 3D model including the point cloud of the user and the exercise equipment;
inputting the IR video of the user, the depth video of the user, and the RGB images of the user into a machine learning model;
estimating a set of joints of the user in the 3D model, the estimating using the point cloud and the machine learning model with the inputted IR video, the inputted depth video, and the inputted RGB images;
tracking, based on the estimated set of joints of the user in the 3D model, the user's motion over a period of time;
determining, based on the tracked user's motion over the period of time and one or more exercise models, a number of repetitions of the exercise performed by the user over the period of time;
determining, based on the tracked user's motion over the period of time and one or more form feedback models, a form feedback value from a set of form feedback values, the set of form feedback values including a satisfactory value and an unsatisfactory value;
calculating, based on the number of repetitions and the form feedback value, a user exercise score;
providing, via a graphical user interface, the user exercise score and the form feedback value to the user, thereby instructing the user to adjust their form during subsequent repetitions of the exercise.
12. The method of claim 11 , further comprising:
identifying, using the one or more RGB sensors, one or more weights associated with the exercise equipment;
tracking position and orientation and motion of the exercise equipment over the period of time;
wherein the form feedback value is determined based the tracked user's motion over the period of time and the tracked position and orientation and motion of the exercise equipment over the period of time, and the one or more form feedback models; and
wherein the user exercise score is calculated based on the one or more weights associated with the exercise equipment and the number of repetitions and the form feedback value.
13. The method of claim 12 , wherein the exercise equipment is of a particular color, and the one or more weights associated with the exercise equipment are identified based on the particular color.
14. The method of claim 11 , wherein the machine learning model comprises a convolutional neural net machine learning model.
15. The method of claim 14 , further comprising:
validating, using another machine learning model and a statistical model, the estimated set of joints of the user in the 3D model prior to tracking the user's motion over the period of time.
16. The method of claim 11 , wherein the point cloud comprises at least 80,000 points, and the estimated set of joints comprises 25 points.
17. The method of claim 11 , wherein the graphical user interface comprises a graphical user interface presented on a screen display of a free-standing A-frame exercise equipment cabinet.
18. The method of claim 12 , further comprising:
ranking the user, based on the user exercise score, relative to a plurality of other users, wherein each of the other users have a corresponding user exercise score;
providing, via the graphical user interface, a leaderboard including a user list, the user list based on the ranking of the user relative to the other users.
19. The method of claim 11 , further comprising:
determining, based on the tracked user's motion over the period of time and one or more other machine learning models, a severity score associated with the exercise performed by the user over the period of time, the severity score indicating a degree of error in some or all of the movements of the exercise performed by the user;
notifying, based on the severity score and a dynamic severity threshold value, the user that the user has made an error.
20. They method of claim 11 , further comprising:
reducing, in response to determining the form feedback value and based on the form feedback value, the number of determined repetitions performed by the user over the period of time.