Method and system for activity classification
An activity classifier system and method that classifies human activities using 2D skeleton data. The system includes a skeleton preprocessor that transforms the 2D skeleton data into transformed skeleton data, the transformed skeleton data comprising scaled, relative joint positions and relative joint velocities. The system also includes a gesture classifier comprising a first recurrent neural network that receives the transformed skeleton data, and is trained to identify the most probable of a plurality of gestures. The system also has an action classifier comprising a second recurrent neural network that receives information from the first recurrent neural networks and is trained to identify the most probable of a plurality of actions.
1 . A method for classifying an activity performed by a person based on an analysis of sequential frames of a video of the person performing the activity, the method comprising:
extracting, for each frame, a skeletal representation that designates locations of a plurality of body parts of the person in that frame and that is indicative of a posture of the person in that frame, so as to generate a sequence of skeletal representations that are collectively indicative of the posture as the person performs the activity;
establishing, based on an analysis of a first skeletal representation in the sequence of skeletal representations, a first most probable gesture from among a plurality of gestures, each of which generalizes movement from one pose to another pose;
establishing, based on an analysis of a second skeletal representation in the sequence of skeletal representations, a second most probable gesture from among the plurality of gestures; and
establishing, based on an analysis of the first and second most probable gestures, a most probable action from among a plurality of actions.
2 . The method of claim 1 , further comprising:
establishing, for each frame, a movement characteristic of each of the plurality of body parts.
3 . The method of claim 2 , wherein said establishing the movement characteristic comprises computing a velocity of each of the plurality of body parts.
4 . The method of claim 1 ,
wherein the first most probable gesture is associated with a first pose in which the person is expected to start and a second pose in which the person is expected to end, and
wherein the first most probable gesture is established by determining that a first one of the sequence of skeletal representations matches the first pose and a third one of the sequence of skeletal representations matches the second pose.
5 . The method of claim 1 , wherein for each skeletal representation, the locations of the plurality of body parts are designated using X and Y coordinates that define two-dimensional (2D) positions of the plurality of body parts in a corresponding frame.
6 . The method of claim 5 , further comprising:
transforming at least some of the 2D positions across the sequence of skeletal representations relative to a given 2D position.
7 . The method of claim 5 , further comprising:
scaling at least some of the 2D positions across the sequence of skeletal representations relative to a given height.
8 . The method of claim 1 , wherein for each skeletal representation, each of the plurality of body parts is represented by a corresponding one of a plurality of dots, which are connected by lines to form a visualization of that skeletal representation.
9 . The method of claim 1 , further comprising:
posting, to an interface, the sequence of skeletal representations, in succession, without the sequential frames.
10 . The method of claim 1 , wherein said extracting is performed in real time as the sequential frames are generated by a camera that captures the person performing the activity.
11 . The method of claim 1 , wherein each of the plurality of body parts corresponds to a different joint.
12 . The method of claim 1 , wherein said establishing of the first and second most probable gestures is performed using a neural network with one or more recurrent units.
13 . The method of claim 1 ,
wherein said extracting, said establishing the first and second most probable gestures, and said establishing the most probable action are performed by a software program that is executing on a computing device that generates the sequential frames, and
wherein the method further comprises:
transmitting the sequential frames and the sequence of skeletal representations to a destination external to the computing device.