Method and system for symmetric recognition of handed activities
This disclosure describes an activity recognition system for asymmetric (e.g., left- and right-handed) activities that leverages the symmetry intrinsic to most human and animal bodies. Specifically, described is 1) a human activity recognition system that only recognizes handed activities but is inferenced twice, once with input flipped, to identify both left- and right-handed activities and 2) a training method for learning-based implementations of the aforementioned system that flips all training instances (and associated labels) to appear left-handed and in doing so, balances the training dataset between left- and right-handed activities.
1. A method of training an activity classifier, the method comprising:
identifying training instances associated with a first handedness, wherein each training instance is associated with a corresponding class label and a corresponding segment of activity data;
for each training instance,
modifying that training instance by—
transforming the corresponding class label to a class label of a second handedness opposite the first handedness, and
transforming the corresponding segment of activity data using a transformation that is associated with the first handedness;
and
training the activity classifier using the modified training instances,
wherein as a result of said modifying, none of the modified training instances are labeled with the first handedness.
2. The method of claim 1 ,
wherein the first handedness is left handedness and the second handedness is right handedness, and
wherein said modifying causes (i) left-handed labels to be replaced with right-handed labels and (ii) the activity data to be horizontally flipped.
3. The method of claim 1 ,
wherein the first handedness is right handedness and the second handedness is left handedness, and
wherein said modifying causes (i) right-handed labels to be replaced with left-handed labels and (ii) the activity data to be horizontally flipped.
4. The method of claim 1 , wherein each corresponding segment of activity data includes a set of keypoints representing skeletal joints of a corresponding person as an array of X and Y coordinates.
5. The method of claim 4 , wherein the transformation is a horizontal transformation that causes each X coordinate to be reversed while leaving each Y coordinate unaltered.
6. The method of claim 1 , wherein each corresponding segment of activity data includes keypoints representing skeletal joints of a corresponding person, images of the corresponding person, or video of the corresponding person.
7. The method of claim 1 , wherein the activity classifier is implemented as a long-short-term memory (LSTM) recurrent neural network that comprises a series of fully connected layers, activation layers, LSTM layers, and softmax layers.
8. The method of claim 7 , wherein at least some of the fully connected layers and the LSTM layers are parameterized by weights that are learned by training on the modified training instances.
9. A method of training a machine learning model, the method comprising:
identifying training instances associated with a first asymmetry, wherein each training instance is associated with a corresponding class label and a corresponding segment of activity data;
for each training instance,
modifying that training instance by—
transforming the corresponding class label to a class label of a second asymmetry opposite the first asymmetry, and
transforming the corresponding segment of activity data; and
training the machine learning model using the modified training instances,
wherein as a result of said modifying, none of the modified training instances are labeled with the first asymmetry.
10. The method of claim 9 ,
wherein each corresponding segment of activity data is transformed by applying a transformation thereto, and
wherein the transformation is associated with the first asymmetry.
11. The method of claim 10 , wherein each corresponding segment of activity data includes a set of keypoints representing skeletal joints of a corresponding person as an array of X and Y coordinates.
12. The method of claim 11 , wherein the transformation is a horizontal transformation that causes each X coordinate to be reversed while leaving each Y coordinate unaltered.
13. The method of claim 11 , wherein the transformation is a vertical transformation that causes each Y coordinate to be reversed while leaving each X coordinate unaltered.
14. The method of claim 9 ,
wherein the first asymmetry is left handedness,
wherein said modifying causes each training instance to be associated with a right-handed label, and
wherein said training causes the machine learning model to be trained to classify right-handed activities represented in the training instances as left-handed activities.
15. The method of claim 9 ,
wherein the first asymmetry is right handedness,
wherein said modifying causes each training instance to be associated with a left-handed label, and
wherein said training causes the machine learning model to be trained to classify left-handed activities represented in the training instances as right-handed activities.
16. The method of claim 9 , wherein the machine learning model is based on a neural network, a decision tree, a support vector machine, or a classifier.
17. The method of claim 9 , wherein the machine learning model is designed to accept, as input, activity data having a same format as the corresponding transformed segments of activity data associated with the modified training instances.
18. A non-transitory medium with instructions stored thereon that, when executed by a processor, cause the processor to perform operations comprising:
identifying training instances associated with a first asymmetry, wherein each training instance is associated with a corresponding class label and a corresponding segment of activity data;
for each training instance,
modifying that training instance by—
transforming the corresponding class label to a class label of a second asymmetry opposite the first asymmetry, and
transforming the corresponding segment of activity data; and
training a machine learning model such that one or more weights are parameterized based on the modified training instances and the corresponding transformed segments of activity data.
19. The non-transitory medium of claim 18 , wherein the machine learning model is implemented as a long-short-term memory (LSTM) neural network that comprises inner product layers, activation layers, LSTM layers, and softmax layers.
20. The non-transitory medium of claim 19 , wherein the inner product layers and the LSTM layers are parameterized by the one or more weights.