Automatic body movement recognition and association system including smoothing, segmentation, similarity, pooling, and dynamic modeling
An automatic body movement recognition and association system that uses two dimensional (2D) and/or three dimensional (3D) skeletal joint information from at least one of a stand-alone depth-sensing image capture device, sensor, wearable sensor, video, and/or video streams that detects the body movements of a user. The automatic body movement recognition and association system can perform various processes on the body movement data, such as smoothing, segmentation, similarity, pooling, and dynamic modeling.
1 . A method for smoothing a sequence of body movement data of a user comprising:
receiving the sequence of body movement data from one of a camera, a video, and a sensor, wherein the sequence of body movement data comprises one of a set of two-dimensional coordinates of a plurality of points of a skeleton of the user and a set of three-dimensional coordinates of a plurality of points of a skeleton of the user;
fine tuning the sequence of body movement data into a smoothed sequence of body movement data on a condition that the sequence does not contain errors;
outputting the smoothed sequence of body movement data;
smoothing the sequence based on a weighted moving average with a smallest window size to determine a first sequence;
determining a reverse sequence based on the sequence;
smoothing the reverse sequence based on the weighted moving average with the smallest window size;
reversing the reverse sequence to determine a second sequence;
determining an average sequence based on the first sequence and the second sequence; and
outputting a fine tuned smoothed sequence based on the average sequence.
2 . A method for segmenting a sequence of body movement data of a user comprising:
receiving the sequence of body movement data from one of a camera, a video, and a sensor, wherein the sequence of body movement data comprises one of a set of two-dimensional coordinates of a plurality of points of a skeleton of the user and a set of three-dimensional coordinates of a plurality of points of a skeleton of the user;
processing, using a preprocessing component, the sequence of body movement data into a preprocessed sequence of body movement data;
determining a sequence curvature across time based on the preprocessed sequence of body movement data;
initializing a curvature model into an initialized curvature model;
fitting initialized curvature model on sequence curvature across time;
labeling a plurality of curvature frames across time based on the initialized curvature model;
initializing a previous gradient direction to a neutral direction;
determining the previous gradient direction and a current gradient direction based on the labeled plurality of curvature frames; and
identifying the plurality of initial segments of body movement data by checking for gradient direction shift from increasing to decreasing gradient.
3 . The method of claim 2 , further comprising:
receiving one of the initial segments of body movement data and the plurality of initial segments of body movement data;
identifying a dynamic forward model based on one of the initial segments of body movement data and the plurality of initial segments of body movement data;
segmenting one of the initial segments of body movement data and the plurality of initial segments of body movement data based on the dynamic forward model; and
identifying one of a segment of body movement data and a plurality of segments of body movement data.