Method and system for symmetric recognition of handed activities
This disclosure describes an activity recognition system for asymmetric (e.g., left- and right-handed) activities that leverages the symmetry intrinsic to most human and animal bodies. Specifically, described is 1) a human activity recognition system that only recognizes handed activities but is inferenced twice, once with input flipped, to identify both left- and right-handed activities and 2) a training method for learning-based implementations of the aforementioned system that flips all training instances (and associated labels) to appear left-handed and in doing so, balances the training dataset between left- and right-handed activities.
1. A non-transitory medium with instructions stored thereon that, when executed by a processor, cause the process-processor to perform operations comprising:
transforming activity data that is generated by a capture device using a horizontal transformation, such that—
activities, if any, that are represented by the activity data and performed on a left side are transformed to a right side, and
activities, if any, that are represented by the activity data and performed on the right side are transformed to the left side;
classifying a given activity based on an analysis of the activity data and then outputting a first activity class for the given activity;
producing, for the given activity, a second activity class of opposite handedness; and
outputting a predicted activity class for the given activity based on which of the first and second activity classes is dominant.
2. The non-transitory medium of claim 1 , wherein said classifying involves applying, to the activity data, an activity classifier that is designed to identify right-handed activities.
3. The non-transitory medium of claim 1 , wherein said classifying involves applying, to the activity data, an activity classifier that is designed to identify left-handed activities.
4. The non-transitory medium of claim 3 , wherein the activity classifier is implemented as a long-short-term memory (LSTM) recurrent neural network that comprises a series of fully connected layers, activation layers, LSTM layers, and softmax layers.
5. The non-transitory medium of claim 1 , wherein said producing involves applying another transformation that corresponds to the horizontal transformation.
6. The non-transitory medium of claim 1 , wherein the horizontal transformation causes each X coordinate to be reversed while leaving each Y coordinate unaltered.
7. The non-transitory medium of claim 1 , wherein the activity data includes a set of keypoints representing skeletal joints of a targeted person as an array of X and Y coordinates.
8. The non-transitory medium of claim 7 ,
wherein the set of keypoints are normalized in a range of zero to one, and
wherein said transforming involves computing, for each X coordinate, an appropriate transformed value by subtracting that X coordinate from one.
9. A method performed by an activity recognition system, the method comprising:
transforming activity data that is generated by a capture device using a symmetric transformation;
classifying a given activity based on an analysis of the activity data and then outputting a first activity class for the given activity;
flipping the first activity class using a transformation that corresponds to the symmetric transformation, so as to produce a second activity class; and
outputting a predicted activity class for the given activity based on which of the first and second activity classes is dominant.
10. The method of claim 9 , wherein the symmetric transformation is a horizontal transformation that causes each X coordinate to be reversed while leaving each Y coordinate unaltered.
11. The method of claim 9 , wherein the symmetric transformation is a vertical transformation that causes each Y coordinate to be reversed while leaving each X coordinate unaltered.
12. The method of claim 9 ,
wherein the activity data is in temporal order and has forward/backward symmetry, and
wherein the symmetric transformation causes the activity data to be reordered in reverse temporal order.
13. The method of claim 9 , wherein the activity data includes keypoints representing skeletal joints of a targeted person, images of the targeted person, or video of the targeted person.
14. The method of claim 9 ,
wherein said classifying involves applying an activity classifier to the activity data, and
wherein the activity classifier is representative of a machine learning model that is parameterized by trainable weights determined via analysis of a class-labeled database of training activity data.
15. The method of claim 14 , wherein the training activity data is in a same format as the activity data generated by the capture device.
16. The method of claim 9 , wherein the predicted activity class is one of multiple predicted activity classes output for the given activity.
17. The method of claim 16 , wherein the multiple predicted activity classes are representative of activity classes for which there is some evidence in either the first activity class or the second activity class.
18. An activity recognition system comprising:
a first module that is configured to transform activity data generated by a capture device using a symmetric transformation;
a second module that is configured to classify, based on an analysis of the activity data, a given activity by outputting a first series of class probabilities;
a third module that is configured to flip the first series of class probabilities using a transformation that corresponds to the symmetric transformation, so as to produce a second series of class probabilities; and
a fourth module that is configured to output either:
(i) a most probable activity class, as determined based on the first and second series of class probabilities, or
(ii) a combined set of class probabilities, as determined based on the first and second series of class probabilities.
19. The activity recognition system of claim 18 , wherein the first module is further configured to receive the activity data from the capture device.
20. The activity recognition system of claim 19 , wherein the activity data includes video that is streamed to the first module from the capture device.