Computer vision-driven interactive full-body motion tracking
Systems and methods described herein relate to computer vision-driven full-body motion tracking using a portable computing device. The portable computing device has a first image sensor and a second image sensor to capture images of a full body of the user while the user is performing exercises. The images are captured while the portable computing device is positioned in a substantially vertical orientation. The images are processed to generate high dynamic range (HDR) image data and to determine depth information associated with the user. Motion tracking data is generated in real time, using the HRD image data and the depth information, while the user is performing an exercise. Real-time interactive feedback may be provided during exercises.
1. One or more non-transitory machine-readable storage media including instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
accessing images of a full body of a user captured by a first image sensor and a second image sensor of a portable computing device, the images captured while the portable computing device is positioned in a substantially vertical orientation and while the user is performing one or more exercises;
processing the images of the first image sensor and the second image sensor to:
generate high dynamic range (HDR) image data, and
determine depth information associated with the user; and
generating, in real time and while the user is performing the one or more exercises, motion tracking data based on the HDR image data and the depth information,
wherein generating the motion tracking data comprises generating pose estimation data by analyzing body landmarks of the user while the user performs the one or more exercises, the pose estimation data obtained by processing first input comprising the HDR image data generated from the images of the first image sensor and the second image sensor and second input comprising the depth information determined from the images of the first image sensor and the second image sensor.
2. The one or more non-transitory machine-readable storage media of claim 1 , wherein the first image sensor and the second image sensor simultaneously capture pairs of images at different exposure levels, and the HDR image data is generated for each pair of images by a process comprising:
aligning a first image from the first image sensor and a second image from the second image sensor to obtain aligned image data, and
generating a final image using the aligned image data.
3. The one or more non-transitory machine-readable storage media of claim 2 , wherein generating the final image comprises performing at least one of exposure fusion or tone mapping.
4. The one or more non-transitory machine-readable storage media of claim 2 , wherein at least one of aligning the first image with the second image or generating the final image comprises executing one or more machine learning models.
5. The one or more non-transitory machine-readable storage media of claim 1 , wherein determining the depth information comprises generating a depth map to estimate a distance between one or more body parts of the user and the portable computing device.
6. The one or more non-transitory machine-readable storage media of claim 1 , wherein determining the depth information comprises executing one or more machine learning models.
7. The one or more non-transitory machine-readable storage media of claim 1 , wherein processing of the images comprises performing stereoscopic correction using a first image from the first image sensor and a second image from the second image sensor.
8. The one or more non-transitory machine-readable storage media of claim 1 , wherein processing the first input comprising the HDR image data and the second input comprising the depth information comprises:
generating the pose estimation data based on the HDR image data; and
adjusting the pose estimation data through post-processing using the depth information.
9. The one or more non-transitory machine-readable storage media of claim 1 , wherein the images comprise a plurality of images captured over time at a fixed, substantially vertical axis.
10. The one or more non-transitory machine-readable storage media of claim 1 , wherein the one or more exercises comprise at least two of:
a first exercise performed in a substantially upright position in front of the portable computing device,
a second exercise performed in a supported position in front of the portable computing device, and
a third exercise performed in a floor-based position in front of the portable computing device,
wherein the portable computing device captures full-body motion of the user without any angular adjustment of the portable computing device between the at least two exercises.
11. The one or more non-transitory machine-readable storage media of claim 1 , wherein each of the first image sensor and the second image sensor has a focal length and sensor size selected to provide:
a diagonal field of view (DFOV) of greater than 110 degrees;
a horizontal field of view (HFOV) of greater than 90 degrees; and
a vertical field of view (VFOV) of greater than 70 degrees.
12. The one or more non-transitory machine-readable storage media of claim 11 , wherein the DFOV is greater than 130 degrees.
13. The one or more non-transitory machine-readable storage media of claim 11 , wherein the HFOV is greater than 100 degrees.
14. The one or more non-transitory machine-readable storage media of claim 11 , wherein the VFOV is greater than 100 degrees.
15. The one or more non-transitory machine-readable storage media of claim 1 , the operations further comprising:
displaying, on a display screen of the portable computing device and while the user is performing the one or more exercises, a user interface comprising a digital video feed representing the images together with at least one of motion tracking output or instructions for performing the one or more exercises.
16. The one or more non-transitory machine-readable storage media of claim 1 , wherein the first image sensor and the second image sensor are front-facing cameras that are spaced apart, along a longitudinal axis of the portable computing device, at a baseline of between 15 mm and 65 mm.
17. The one or more non-transitory machine-readable storage media of claim 1 , wherein the portable computing device is a substantially rectangular tablet computing device, and the first image sensor and the second image sensor are front-facing cameras positioned at or adjacent a long edge of the portable computing device.
18. The one or more non-transitory machine-readable storage media of claim 1 , wherein the images are captured while a support stand holds a front face of the portable computing device in the substantially vertical orientation.
19. A computer-implemented method comprising:
accessing images of a full body of a user captured by a first image sensor and a second image sensor of a portable computing device, the images captured while the portable computing device is positioned in a substantially vertical orientation and while the user is performing one or more exercises;
processing the images of the first image sensor and the second image sensor to:
generate high dynamic range (HDR) image data, and
determine depth information associated with the user; and
generating, in real time and while the user is performing the one or more exercises, motion tracking data based on the HDR image data and the depth information,
wherein generating the motion tracking data comprises generating pose estimation data by analyzing body landmarks of the user while the user performs the one or more exercises, the pose estimation data obtained by processing first input comprising the HDR image data generated from the images of the first image sensor and the second image sensor and second input comprising the depth information determined from the images of the first image sensor and the second image sensor.
20. The one or more non-transitory machine-readable storage media of claim 1 , wherein the body landmarks comprise three-dimensional (3D) landmark positions, and generating the motion tracking data comprises:
while the user performs the one or more exercises, executing a machine learning model that processes the HDR image data as the first input and the depth information as the second input to generate the 3D landmark positions.