Video-based motion counting and analysis systems and methods for virtual fitness application
A system and method for implementing a virtual fitness application are disclosed. Embodiments may be built for various platforms, including web browsers and mobile phones. One embodiment uses pose estimation to count the repetitive motions of a fitness activity (e.g., running). In one embodiment, to count the repetitive motions, a repetitive motion counting process is used that is based on computing differences of Y-coordinates of key points of the user. The repetitive motion counting process includes selecting a proper pose of the user; computing one or more delta values (corresponding to Y-coordinate changes of the key points); and counting a given user movement as a repetitive motion based on a function (e.g., average) of the delta values. One embodiment uses the pose estimation and repetitive motion count results to provide a gamified experience, for example, a leaderboard, a stats report, instant visual feedback, badges, coins, and a social experience.
1. A computer-implemented method executable by a hardware processor for implementing a single-user virtual fitness application, comprising:
receiving an input video of one or more persons from a camera;
detecting a sequence of human poses in the input video using a pose estimation process, wherein the pose estimation process is based on a Convolutional Neural Network (CNN);
selecting a proper pose for a main user of the virtual fitness application, when the hardware processor has detected multiple poses in a given frame of the input video, to generate a sequence of proper poses;
detecting one or more key points in the sequence of proper poses, the one or more key points having Y-coordinates;
computing one or more delta values (y t -y t-1 ) corresponding to changes in the Y-coordinates of the one or more key points;
computing a function of the delta values of the one or more key points in the sequence of proper poses; and
counting a given user movement as a repetitive motion of a fitness activity based on the function of the delta values; and
computing a plurality of statistics about the fitness activity based on the counting.
2. The computer-implemented method of claim 1 ,
wherein the fitness activity is selected from the group consisting of running, jogging, walking, jumping, performing jumping jacks, squatting, and dribbling, and
wherein the one or more repetitive motions are selected from the group consisting of steps, jumps, squats, and dribbles.
3. The computer-implemented method of claim 1 , wherein the selecting the proper pose comprises:
selecting a most centered pose as the proper pose, when the hardware processor has detected multiple poses in a given frame.
4. The computer-implemented method of claim 1 , wherein the selecting the proper pose comprises:
selecting the proper pose utilizing a human tracking algorithm, when the hardware processor has detected multiple poses in a given frame.
5. The computer-implemented method of claim 1 , further comprising:
presenting one or more gamification elements based on the plurality of statistics.
6. The computer-implemented method of claim 1 , wherein the input video is captured using the camera selected from the group consisting of a mobile device camera and a portable camera device.
7. The computer-implemented method of claim 1 , wherein the one or more key points are selected from the group consisting of a body joints, a nose, an eyes, an ears, a chest, and a shoulders of the main user.
8. The computer-implemented method of claim 1 , wherein the function of the delta values is selected from the group consisting of a mean, a median, and a single delta value selection.
9. The computer-implemented method of claim 1 , wherein the function of the delta values is a mean delta value, and wherein the method further comprises:
counting the given user movement as the repetitive motion when the mean delta value changes from positive, to negative, to positive.
10. The computer-implemented method of claim 1 , further comprising:
applying a smoothing function on the Y-coordinates of the one or more key points before computing the delta values.
11. The computer-implemented method of claim 1 , further comprising:
performing a checking process on the given user movement's metrics to invalidate the given user movement based on one or more criteria.
12. The computer-implemented method of claim 11 , further comprising
excluding the given user movement when its rising period is more than a given threshold; and
excluding the given user movement when its rising amplitude is smaller than a given threshold.
13. The computer-implemented method of claim 11 , further comprising:
adjusting the checking process dynamically based on a detection sensitivity parameter.
14. The computer-implemented method of claim 11 , further comprising:
utilizing a limb movement to control the checking process, wherein the limb movement is selected from the group consisting of a hand swing and a leg movement.
15. A non-transitory storage medium storing program code for implementing a single-user virtual fitness application, the program code executable by a hardware processor, the program code when executed by the hardware processor causes the hardware processor to:
receive an input video of one or more persons from a camera;
detect a sequence of human poses in the input video using a pose estimation process, wherein the pose estimation process is based on a Convolutional Neural Network (CNN);
select a proper pose for a main user of the virtual fitness application, when the hardware processor has detected multiple poses in a given frame of the input video, to generate a sequence of proper poses;
detect one or more key points in the sequence of proper poses, the one or more key points having Y-coordinates;
compute one or more delta values (y t -y t-1 ) corresponding to changes in the Y-coordinates of the one or more key points;
compute a function of the delta values of the one or more key points in the sequence of proper poses; and
count a given user movement as a repetitive motion of a fitness activity based on the function of the delta values; and
compute a plurality of statistics about the fitness activity based on the counting.
16. The non-transitory storage medium of claim 15 ,
wherein the fitness activity is selected from the group consisting of running, jogging, walking, jumping, performing jumping jacks, squatting, and dribbling, and
wherein the one or more repetitive motions are selected from the group consisting of steps, jumps, squats, and dribbles.