Exercise guidance using multi-modal data
A first output is received from a first hardware optical sensor. A second output is received from a second hardware sensor. Guidance is provided for a movement via a user interface, wherein the guidance is based at least in part on: the first output from the first hardware optical sensor; the second output from the second hardware sensor; and a model based at least in part on historical performance of the movement; and wherein at least one of the first output and the second output triggers a condition.
1. An exercise system, comprising:
a first hardware optical sensor;
a second hardware sensor;
a user interface that provides guidance for a movement, wherein the user interface comprises a display on the exercise system and wherein the guidance is based at least in part on:
a first output from the first hardware optical sensor for a user;
a second output from the second hardware sensor for the user;
a position data based at least in part on the second output that is associated with a cable position for a cable coupled to an actuator for the user;
a pose data model for the user based at least in part on historical performance of the movement;
wherein the pose data model comprises pose data generated at least in part on the first output;
wherein pose data comprises a set of canonical key points each positioned in three-dimensional space at a specific time and wherein each canonical key point of the set of canonical key points represents a joint for the user;
a sensor fusion data comprising the position data combined with the pose data model via preprocessing;
wherein the sensor fusion data is based at least in part on a synchronization of the position data with the pose data model based at least in part on a reference timestamp;
a trigger variable value determined at least in part on the sensor fusion data; and
a prediction triggered at least in part on the trigger variable value; and
wherein the prediction provides the guidance for the movement via the display in an event a confidence parameter for the prediction is above a threshold.
2. The system of claim 1 , wherein the historical performance of the movement is sensed at least in part by the first hardware optical sensor and the second hardware sensor.
3. The system of claim 1 , wherein the condition is a predetermined condition.
4. The system of claim 1 , wherein the first hardware optical sensor is at least one of the following: a camera, a mobile camera, a front-facing camera, a side-facing camera, an overhead camera, a lidar sensor, a radar sensor, a depth-sensing sensor, an infrared positioning/reckoning sensor, an imaging stereo depth positioning/reckoning sensor, a near-field positioning/reckoning sensor, and a sonar sensor.
5. The system of claim 1 , wherein the second hardware sensor is at least one of the following: a cable position sensor, a motor position sensor, an IMU, an accelerometric sensor, a gyroscopic sensor, a camera, a mobile camera, a front-facing camera, a side-facing camera, an overhead camera, a lidar sensor, a radar sensor, a depth-sensing sensor, an infrared positioning/reckoning sensor, an imaging stereo depth positioning/reckoning sensor, a near-field positioning/reckoning sensor, and a sonar sensor.
6. The system of claim 1 , wherein guidance comprises at least one of the following: form feedback, mobility assessment, stability assessment, repetition counting, determining ROM (range of motion), determining flexibility, determining strength, struggle detection, fatigue detection, and/or balance assessment.
7. The system of claim 1 , wherein the model comprises a transformation, wherein the transformation comprises at least one of the following: a video transform to pose data key points, a position transform to ROM, a position transform to speed, and a position transform to tension.
8. The system of claim 1 , wherein the model comprises a featuring engineer for trigger variables.
9. The system of claim 1 , wherein the model comprises a classifier to make a first trigger variable from the first output of the first hardware optical sensor.
10. The system of claim 1 , wherein the model comprises a classifier to make a second trigger variable from the second output of the second hardware sensor.
11. The system of claim 1 , wherein the model comprises a synchronization of a first trigger variable based at least in part on a reference time for a second trigger variable.
12. The system of claim 1 , wherein the model comprises a synchronization of a first trigger variable with a second trigger variable based at least in part on a time window.
13. The system of claim 1 , wherein the model comprises a synchronization of a first trigger variable with a second trigger variable based at least in part on smoothing, rep detection, and/or interpolation in the event the first output lags.
14. The system of claim 1 , wherein the model comprises trigger rules combining a first trigger variable and a second trigger variable, wherein the trigger rules comprise at least one of the following: machine learning trigger rules and algorithmic trigger rules.
15. The system of claim 14 , wherein the combining comprises at least one of the following: logical combining, Boolean combining, algebraic combining, trigonometric combining, weighted combining, threshold combining, binary combining, continuous combining, and mathematical combining.
16. The system of claim 1 , wherein the model comprises an output of guidance score or guidance feedback to the user interface at least in part to provide guidance for the movement.
17. A method, comprising:
receiving a first output from a first hardware optical sensor;
receiving a second output from a second hardware sensor; and
providing guidance for a movement via a user interface, wherein the user interface comprises a display on the exercise system and wherein the guidance is based at least in part on:
the first output from the first hardware optical sensor for a user;
the second output from the second hardware sensor for the user;
a position data based at least in part on the second output that is associated with a cable position for a cable coupled to an actuator for the user;
a pose data model for the user based at least in part on historical performance of the movement;
wherein the pose data model comprises pose data generated at least in part on the first output;
wherein pose data comprises a set of canonical key points each positioned in three-dimensional space at a specific time and wherein each canonical key point of the set of canonical key points represents a joint for the user;
a sensor fusion data comprising the position data combined with the pose data model via preprocessing;
wherein the sensor fusion data is based at least in part on a synchronization of the position data with the pose data model based at least in part on a reference timestamp;
a trigger variable value determined at least in part on the sensor fusion data; and
a prediction triggered at least in part on the trigger variable value; and
wherein prediction provides the guidance for the movement via the display in an event a confidence parameter for the prediction is above a threshold.
18. The method of claim 1 , wherein the historical performance of the movement is sensed at least in part by the first hardware optical sensor and the second hardware sensor.
19. A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:
receiving a first output from a first hardware optical sensor;
receiving a second output from a second hardware sensor; and
providing guidance for a movement via a user interface, wherein the user interface comprises a display on the exercise system and wherein the guidance is based at least in part on:
the first output from the first hardware optical sensor for a user;
the second output from the second hardware sensor for the user;
a position data based at least in part on the second output that is associated with a cable position for a cable coupled to an actuator for the user;
a pose data model for the user based at least in part on historical performance of the movement;
wherein the pose data model comprises pose data generated at least in part on the first output;
wherein pose data comprises a set of canonical key points each positioned in three-dimensional space at a specific time and wherein each canonical key point of the set of canonical key points represents a joint for the user;
a sensor fusion data comprising the position data combined with the pose data model via preprocessing;
wherein the sensor fusion data is based at least in part on a synchronization of the position data with the pose data model based at least in part on a reference timestamp;
a trigger variable value determined at least in part on the sensor fusion data; and
a prediction triggered at least in part on the trigger variable value; and
wherein prediction provides the guidance for the movement via the display in an event a confidence parameter for the prediction is above a threshold.
20. The computer program product of claim 19 , wherein the historical performance of the movement is sensed at least in part by the first hardware optical sensor and the second hardware sensor.