Skeletal tracking using previous frames

1. A method comprising:

receiving, by one or more processors, a monocular image that includes a depiction of a body of a user;

detecting, by the one or more processors, a plurality of skeletal joints of the body based on the monocular image;

identifying a plurality of skeletal joint features of the monocular image using a first machine learning technique, wherein positions of the plurality of skeletal joints are detected based on the identified plurality of skeletal joint features;

accessing a video feed comprising a plurality of monocular images received prior to the monocular image;

filtering, using the video feed, the plurality of skeletal joints of the body detected based on the monocular image; and

determining a pose represented by the body depicted in the monocular image based on the filtered plurality of skeletal joints of the body.

2. The method of claim 1 , further comprising:

modifying, by the one or more processors, a pose of an avatar to match the pose represented by the body depicted in the monocular image by adjusting a set of skeletal joints of a rig of an avatar based on the filtered plurality of skeletal joints of the body; and

generating, for display by the one or more processors, the avatar having t modified pose that matches the pose represented by the body depicted in the monocular image.

3. The method of claim 1 , wherein the monocular image is a first frame of a video.

4. The method of claim 3 , wherein the first machine learning technique comprises a first deep neural network.

5. The method of claim 4 , further comprising training the first deep neural network by performing operations comprising:

receiving training data comprising a plurality of training monocular images and ground truth skeletal joint information for each of the plurality of training monocular images, each of the plurality of training monocular images depicting a different body pose;

applying the first deep neural network to a first training monocular image of the plurality of training monocular images to estimate skeletal joints of the body depicted in the first training monocular image;

computing a deviation between the estimated skeletal joints of the body and the ground truth skeletal joint information associated with the first training monocular image;

updating parameters of the first deep neural network based on the computed deviation; and

repeating the applying, computing and updating steps for each of the plurality of training monocular images.

6. The method of claim 1 , wherein filtering the plurality of skeletal joints comprises applying the video teed to a second machine learning technique to estimate skeletal joint positions.

7. The method of claim 6 , wherein the second machine learning technique comprises a second deep neural network, and wherein the filtering comprises comparing a prediction of the skeletal joint positions for the monocular image provided by the second deep neural network with the plurality of skeletal joint features of the monocular image identified using a first machine learning technique.

8. The method of claim 7 , further comprising training the second deep neural network by performing operations comprising:

receiving training data comprising a plurality of training videos and ground truth skeletal joint information for each of the plurality of training videos, each of the plurality of training videos depicting a different body pose;

applying the second deep neural network to a first training video of the plurality of training videos to predict skeletal joints of the body in a frame subsequent to the first training video;

computing a deviation between the predicted skeletal joints of the body and the ground truth skeletal joint information associated with the first training video;

updating parameters of the second deep neural network based on the computed deviation; and

repeating the applying, computing and updating steps for each of the plurality of training videos.

9. The method of claim 1 , wherein the plurality of monocular images was received a threshold number of seconds prior to receiving the monocular image.

10. The method of claim 1 further comprising selecting; by the one or more processors, an avatar associated with a rig from a plurality of avatars.

11. The method of claim 1 further comprising:

receiving a second video comprising a plurality of monocular images that include the depiction of the body of the user;

tracking changes in the plurality of skeletal joints across the plurality of monocular images;

detecting changes to the pose represented by the body based on tracking the changes in the plurality of skeletal joints; and

continuously or periodically modifying poses of an avatar to match the changes to the pose represented by the body.

12. The method of claim 1 further comprising causing an avatar to interact with a virtual object depicted in a given image based on the determined pose.

13. The method of claim 1 , wherein the detecting, filtering, and the determining steps are performed without accessing depth information from a depth sensor.

14. The method of claim 1 , wherein detecting the plurality of skeletal joints of the body comprises identifying points respectively associated with a right wrist, a right elbow, a right shoulder, a nose on a face of the user, a left shoulder, a left elbow, and a left wrist.

15. The method of claim 1 , wherein a rate at which the plurality of skeletal joints is detected is adjusted based on a position of the user relative to an image capture device.

16. A system comprising:

a processor configured to perform operations comprising:

receiving a monocular image that includes a depiction of a body of a user;

detecting a plurality of skeletal joints of the body based on the monocular image;

identifying a plurality of skeletal joint features of the monocular image using a first machine learning technique, wherein positions of the plurality of skeletal joints are detected based on the identified plurality of skeletal joint features;

accessing a video feed comprising a plurality of monocular images received prior to the monocular image;

filtering, using the video feed, the plurality of skeletal joints of the body detected based on the monocular image; and

determining a pose represented by the body depicted in the monocular image based on the filtered plurality of skeletal joints of the body.

17. The system of claim 16 , wherein the operations further comprise filtering the plurality of skeletal joints by applying the video feed to a second machine learning technique to estimate skeletal joint positions.

18. The system of claim 17 , wherein the second machine learning technique comprises a deep neural network, and wherein the operations further comprise training the second machine learning technique by:

receiving training data comprising a plurality of training videos and ground truth skeletal joint information for each of the plurality of training videos, each of the plurality of training videos depicting a different body pose;

applying the deep neural network to a first training video of the plurality of training videos to estimate skeletal joints of the body depicted in the first training video;

computing a deviation between the estimated skeletal joints of the body and the ground truth skeletal joint information associated with the first training video;

updating parameters of the deep neural network based on the computed deviation; and

repeating the applying, computing and updating steps for each of the plurality of training videos.

19. A non-transitory machine-readable storage medium that includes instructions that, when executed by one or more processors of a machine; cause the machine to perform operations comprising:

receiving a monocular image that includes a depiction of a body of a user;

detecting a plurality of skeletal joints of the body based on the monocular image;

identifying a plurality of skeletal joint features of the monocular image using a first machine learning technique, wherein positions of the plurality of skeletal joints are detected based on the identified plurality of skeletal joint features;

accessing a video feed comprising a plurality of monocular images received prior to the monocular image;

filtering; using the video feed, the plurality of skeletal joints of the body detected based on the monocular image; and

determining a pose represented by the body depicted in the monocular image based on the filtered plurality of skeletal joints of the body.

20. The non-transitory machine-readable medium of claim 19 , wherein the operations further comprise filtering the plurality of skeletal joints by applying the video feed to a second machine learning technique to estimate skeletal joint positions.