Systems and methods for predicting lower body poses
A computing system may receive sensor data from one or more sensors coupled to a user. Based on sensor data, the computing system may generate an upper body pose that corresponds to a first portion of a body of the user, which comprises a head and an arm of the user. The computing system may determine contextual information associated with the user. The computing system may generate a lower body pose corresponding to a second portion of the body of the user comprising a leg of the user based on the upper body pose and the contextual information. The computing system may generate a full body pose of the user based on the first upper body pose and the lower body pose.
1. A method comprising, by a computing system:
receiving sensor data captured by one or more sensors coupled to a user;
generating, based on the sensor data, an upper body pose that corresponds to a first portion of a body of the user, the first portion of the body comprising a head and an arm of the user;
determining contextual information associated with the user;
generating, based on the upper body pose and the contextual information, a lower body pose that corresponds to a second portion of the body of the user, the second portion of the body comprising a leg of the user; and
generating a full body pose of the user based on the upper body pose and the lower body pose.
2. The method of claim 1 , wherein generating the lower body pose is further based on a machine-learning model, and wherein the machine-learning model is trained using a second machine-learning model trained to determine whether a given full body pose is likely generated using the machine-learning model.
3. The method of claim 2 , wherein the machine-learning model is optimized during training to cause a second machine-learning model to incorrectly determine that a given full body pose, generated using the machine-learning model, is unlikely generated using the machine-learning model.
4. The method of claim 1 , wherein the one or more sensors are associated with a head-mounted device worn by the user.
5. The method of claim 1 , wherein generating the upper body pose comprises:
determining, based on the sensor data, a plurality of poses corresponding to a plurality of predetermined body parts of the user; and
inferring the upper body pose based on the plurality of poses.
6. The method of claim 5 , wherein the upper body pose further comprises a wrist pose corresponding to a wrist of the user, wherein the wrist pose is determined based on one or more images captured by a head-mounted device worn by the user, the one or more images depicting (1) the wrist of the user or (2) a device held by the user.
7. The method of claim 1 , wherein generating the lower body pose comprises processing the upper body pose and the contextual information using a machine-learning model.
8. The method of claim 7 , wherein the contextual information comprises an application the user is interacting with when the upper body pose is determined.
9. The method of claim 7 , wherein the contextual information indicates the user is standing or sitting at a first time when the sensor data was captured.
10. The method of claim 1 , further comprising generating, based on the full body pose, an avatar of the user.
11. The method of claim 1 , wherein the lower body pose is generated by further processing one or more physical constraints associated with the upper body pose using a machine-learning model.
12. The method of claim 11 , wherein the one or more physical constraints comprise a physical limitation on a range of motion of one or more joints of the user.
13. One or more computer-readable non-transitory non-volatile storage media comprising software that is operable when executed by a server to:
receive sensor data captured by one or more sensors coupled to a user;
generate, based on the sensor data, an upper body pose that corresponds to a first portion of a body of the user, the first portion of the body comprising a head and an arm of the user;
determine contextual information associated with the user;
generate, based on the upper body pose and the contextual information, a lower body pose that corresponds to a second portion of the body of the user, the second portion of the body comprising a leg of the user; and
generate a full body pose of the user based on the upper body pose and the lower body pose.
14. The media of claim 13 , wherein generating the upper body pose comprises:
determining, based on the sensor data, a plurality of poses corresponding to a plurality of predetermined body parts of the user; and
inferring the upper body pose based on the plurality of poses.
15. The media of claim 13 , wherein generating the lower body pose comprises processing the upper body pose and the contextual information using a machine-learning model.
16. The media of claim 15 , wherein the contextual information comprises an application the user is interacting with when the upper body pose is determined.
17. The media of claim 13 , wherein the software is further operable when executed to generate, based on the full body pose, an avatar of the user.
18. The media of claim 13 , wherein the lower body pose is generated by further processing one or more physical constraints associated with the upper body pose using a machine-learning model.
19. A system comprising:
one or more processors; and
one or more computer-readable non-transitory non-volatile storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to:
receive sensor data captured by one or more sensors coupled to a user;
generate, based on the sensor data, an upper body pose that corresponds to a first portion of a body of the user, the first portion of the body comprising a head and an arm of the user;
determine contextual information associated with the user;
generate, based on the upper body pose and the contextual information, a lower body pose that corresponds to a second portion of the body of the user, the second portion of the body comprising a leg of the user; and
generate a full body pose of the user based on the upper body pose and the lower body pose.
20. The system of claim 19 , wherein generating the upper body pose comprises:
determining, based on the sensor data, a plurality of poses corresponding to a plurality of predetermined body parts of the user; and
inferring the upper body pose based on the plurality of poses.