Repetition counting and classification of movements systems and methods

1. A method for determining a number of repetitive motions in a video, the method executed by a processor, the method comprising:

determining a plurality of images from a video, wherein the plurality of images are segmented from at least one video frame of the video;

performing a landmark detection process on a feature of the plurality of images to generate one or more landmarks;

determining one or more principal component axes on a plurality of points associated with a given landmark of the one or more landmarks; and

determining at least one repetitive motion based on a pattern associated with a projection of the plurality of points onto the principal component axes, wherein the determining the repetitive motion is based on a number of zero-crossing pairs associated with the projection of the plurality of points onto the one or more principal component axes.

2. The method of claim 1 , wherein the plurality of images comprise a first image and a second image, wherein the performing the landmark detection process respectively generates a first landmark and a second landmark, wherein the determining the at least one repetitive motion further comprises determining a first pattern and a second pattern, and wherein the method further comprises:

determining a deduplicated pattern based on a comparison of the first pattern and the second pattern; and

determining the at least one repetitive motion based on the deduplicated pattern.

3. The method of claim 1 , wherein the determining the repetitive motion based on the number of zero-crossing pairs further comprises:

generating a distance time series data vector based on the projection of the plurality of points onto the principal component axes;

generating a second time series data based on a determination of the maxima of the distance time series data vector;

determining zero crossings of the second time series data; and

determining the number of zero-crossing pairs having associated opposing directions based on the zero crossings.

4. The method of claim 3 , wherein the generating the second time series data based on the maxima of the distance time series data vector further comprises:

maximizing a function comprising a linear combination of components of the distance time series data vector.

5. The method of claim 3 , wherein the determining the zero crossings of the second time series data further comprises:

performing a debouncing process, wherein a first deviation from a first zero crossing and a second deviation from a second zero crossing have a difference in magnitude below a predetermined threshold, and wherein the first zero crossing and the second zero crossing are determined as a same zero crossing.

6. The method of claim 1 , wherein the performing the landmark detection process further comprises:

determining landmark points identifying a given landmark's motion and a duration parameter associated with a duration of the at least one repetitive motion; and

inputting the landmark points and the duration parameter to a machine learning algorithm to obtain a classification result of the repetitive motion, the classification result identifying the repetitive motion into a predetermined type.

7. The method of claim 6 , wherein the machine learning algorithm comprises a multi-layer neural network.

8. The method of claim 1 , wherein the video is captured by a camera on a mobile device.

9. The method of claim 1 , wherein the feature comprises at least one of a moving object and at least a portion of a body of a moving human.

10. A non-transitory computer-readable medium storing computer-executable instructions which, when executed by a processor, cause the processor to perform operations for determining a number of repetitive motions in a video, the operations comprising:

determine a plurality of images from a video, wherein the plurality of images are segmented from at least one video frame of the video;

perform a landmark detection process on a feature of the plurality of images to generate one or more landmarks;

determine one or more principal component axes on a plurality of points associated with a given landmark of the one or more landmarks; and

determine at least one repetitive motion based on a pattern associated with a projection of the plurality of points onto the principal component axes, wherein the determining the repetitive motion is based on a number of zero-crossing pairs associated with the projection of the plurality of points onto the one or more principal component axes.

11. The non-transitory computer-readable medium of claim 10 , wherein the plurality of images comprise a first image and a second image, wherein the operation to perform the landmark detection process respectively generates a first landmark and a second landmark, wherein the operation to determine the repetitive motion further determines a first pattern and a second pattern, and wherein the operations further comprise:

determine a deduplicated pattern based on a comparison of the first pattern and the second pattern; and

determine the at least one repetitive motion based on the deduplicated pattern.

12. The non-transitory computer-readable medium of claim 10 , wherein the operation to determine the repetitive motion based on the number of zero-crossing pairs further comprises operations to:

generate a distance time series data vector based on the projection of the points onto the principal component axes;

generate a second time series data based on a determination of the maxima of the distance time series data vector;

determine zero crossings of the second time series data; and

determine the number of zero-crossing pairs having associated opposing directions based on the zero crossings.

13. The non-transitory computer-readable medium of claim 12 , wherein the operation to generate the second time series data based on the maxima of the distance time series data vector further comprises operations to:

maximize a function comprising a linear combination of components of the distance time series data vector.

14. The non-transitory computer-readable medium of claim 12 , wherein the operation to determine the zero crossings of the second time series data further comprises operations comprising:

perform a debouncing process, wherein a first deviation from a first zero crossing and a second deviation from a second zero crossing have a difference in magnitude below a predetermined threshold, and wherein the first zero crossing and the second zero crossing are determined as a same zero crossing.

15. The non-transitory computer-readable medium of claim 10 , wherein the computer-executable instructions to perform the landmark detection process further comprises operations comprising:

determine landmark points identifying a given landmark's motion and a duration parameter associated with a duration of the at least one repetitive motion; and

input the points and the duration parameter to a machine learning algorithm to obtain a classification result of the repetitive motion, the classification result identifying the repetitive motion into a predetermined type.

16. The non-transitory computer-readable medium of claim 15 , wherein the machine learning algorithm comprises a multi-layer neural network.

17. The non-transitory computer-readable medium of claim 10 , wherein the video is captured by a camera on a mobile device.

18. The non-transitory computer-readable medium of claim 10 , wherein the feature comprises at least one of a moving object and at least a portion of a body of a moving human.

19. A device for determining a number of repetitive motions in a video, comprising:

a camera and a processor, wherein the processor is configured to:

determine a plurality of images from a video, wherein the plurality of images are segmented from at least one video frame of the video;

perform a landmark detection process on a feature of the plurality of images to generate one or more landmarks;

determine one or more principal component axes on a plurality of points associated with a given landmark of the one or more landmarks; and

determine at least one repetitive motion based on a pattern associated with a projection of the plurality of points onto the principal component axes, wherein determining the repetitive motion is based on a number of zero-crossing pairs associated with the projection of the plurality of points onto the principal component axes.

20. The device of claim 19 , wherein the plurality of images comprise a first image and a second image, wherein the processor is configured to generate a first pose landmark and a second pose landmark, wherein the processor is configured to determine a first pattern and a second pattern, and wherein the processor is further configured to:

determine a deduplicated pattern based on a comparison of the first pattern and the second pattern; and

determine the at least one repetitive motion based on the deduplicated pattern.