System and method for vision-based joint action and pose motion forecasting
A computer-implemented method, comprising extracting each frame of a video sequence, estimating current poses of a subject within each frame and determining joint locations for a joint associated with the subject within each frame, computing optical flows between each pair of consecutive video frames, extracting motion features from each video frame of the video sequence based on the optical flows, encoding state information based on the current poses and the motion features, determining a current action label for each motion feature in the state information for a first frame; predicting future action labels for each motion feature in a second frame subsequent to the first frame, based on the current pose, action label and state information, predicting future poses for each motion feature in the second frame based on the current poses and the state information; and refining the current action label, the future action labels, and the future poses.
1. A computer-implemented method, comprising:
extracting each frame of a video sequence at a single frame rate, the video sequence including a series of video frames;
estimating current poses of a subject within each video frame and determining joint locations for a joint associated with the subject within each video frame;
computing optical flows between each pair of consecutive video frames for each time step of the video sequence;
extracting motion features from each video frame of the video sequence based on the optical flows; and
encoding state information based on the current poses and the motion features.
2. The method of claim 1 further comprising:
determining a current action label for each motion feature contained in the state information for a first frame of the series of video frames, the series of video frames being in red green blue (RGB) format;
predicting, by a decoder, future action labels for each motion feature in a second frame of the series of video frames subsequent to the first frame, based on the current pose, action label and the state information;
predicting, by a decoder, future poses for each motion feature in the second frame based on the current poses and the state information; and
refining the current action label, the future action labels, and the future poses based on a loss function.
3. The method of claim 2 further comprising predicting at least one future pose and future motion for the second frame based on the model and the video sequence.
4. The method of claim 1 wherein the encoder and the decoder comprise one or more processors.
5. The method of claim 4 , wherein the encoder and the decoder are implemented using a recurrent neural network, implementing one or more gated recurrent network.
6. The method of claim 1 wherein the computing the optical flows forms two-channel flow frames where each channel contains displacements at x and y axis, respectively.
7. The method of claim 1 wherein the frame rate is between 24 fps and 60 fps.
8. The method of claim 1 wherein each joint location of the one or more joint locations comprises a pair of two dimensional coordinates (e.g., X,Y) or 3D coordinates (e.g., X,Y,Z).
9. A non-transitory computer readable medium including instructions executable on a processor, the instructions comprising:
extracting, by an encoder, each frame of a video sequence at a single frame rate, the video sequence including a series of video frames;
estimating current poses of one or more subject within each video frame and determining joint locations for a joint associated with the subject within each video frame the video sequence;
computing optical flows between each pair of consecutive video frames each time step of the video sequence;
extracting motion features from each video frame of the video sequence based on the optical flows; and
encoding, by an encoder, state information based on the current poses and the motion features.
10. The non-transitory computer readable medium of claim 9 further comprising:
determining a current action label for each motion feature contained in the state information for a first frame of the series of video frames;
predicting, by a decoder, future action labels for each motion feature in a second frame of the series of video frames subsequent to the first frame, based on the current pose, action label and the state information;
predicting, by a decoder, future poses for each motion feature in the second frame based on the current poses and the state information; and
refining the current action label, the future action labels, and the future poses based on a loss function.
11. The non-transitory computer readable medium of claim 10 further comprising predicting at least one future pose and future motion for the second frame based on the model and the video sequence.
12. The non-transitory computer readable medium of claim 9 wherein the encoder and the decoder comprise one or more processors, and the encoder and the decoder are implemented using a recurrent neural network, implementing one or more gated recurrent network.
13. The non-transitory computer readable medium of claim 9 wherein the computing the optical flows forms two-channel flow frames where each channel contains displacements at x and y axis, respectively.
14. The non-transitory computer readable medium of claim 9 wherein the frame rate is between 24 fps and 60 fps.
15. The non-transitory computer readable medium of claim 9 wherein each joint location of the one or more joint locations comprises a pair of two dimensional coordinates (e.g., X,Y) or 3D coordinates (e.g., X,Y,Z).
16. A system for vision-based joint action and pose motion forecasting including a processor and a storage, the system comprising:
an encoder configured to extract each frame of a video sequence at a single frame rate, the video sequence including a series of video frames;
the processor estimating current poses of one or more subject within each video frame and determining joint locations for one or more joints associated with one or more subjects within each video frame the video sequence;
the processor computing optical flows between each pair of consecutive video frames each time step of the video sequence;
the processor extracting motion features from each video frame of the video sequence based on the optical flows; and
an encoder encoding state information based on the current poses and the motion features.
17. The system of claim 16 , where the processor is configured to perform:
determining a current action label for each motion feature contained in the state information for a first frame of the series of video frames;
predicting, by a decoder, future action labels for each motion feature in a second frame of the series of video frames subsequent to the first frame, based on the current pose, action label and the state information;
predicting, by a decoder, future poses for each motion feature in the second frame based on the current poses and the state information;
refining the current action label, the future action labels, and the future poses based on a loss function; and
predicting at least one future pose and future motion for the second frame based on the model and the video sequence.
18. The system of claim 16 wherein the encoder and the decoder comprise one or more processors, and wherein the encoder and the decoder are implemented using a recurrent neural network, implementing one or more gated recurrent network.
19. The system of claim 16 wherein the processor computing the optical flows forms two-channel flow frames where each channel contains displacements at x and y axis, respectively.
20. The system of claim 16 wherein each joint location of the one or more joint locations comprises a pair of two dimensional coordinates (e.g., X,Y) or 3D coordinates (e.g., X,Y,Z).