Machine-learning based gesture recognition using multiple sensors
A device implementing a system for machine-learning based gesture recognition includes at least one processor configured to, receive, from a first sensor of the device, first sensor output of a first type, and receive, from a second sensor of the device, second sensor output of a second type that differs from the first type. The at least one processor is further configured to provide the first sensor output and the second sensor output as inputs to a machine learning model, the machine learning model having been trained to output a predicted gesture based on sensor output of the first type and sensor output of the second type. The at least one processor is further configured to determine the predicted gesture based on an output from the machine learning model, and to perform, in response to determining the predicted gesture, a predetermined action on the device.
1. A method comprising:
receiving, from a first sensor of a device, first sensor output of a first type;
receiving, from a second sensor of the device, second sensor output of a second type that differs from the first type;
providing the first sensor output and the second sensor output as inputs to a machine learning model, the machine learning model having been trained to output a predicted gesture based on sensor output of the first type and sensor output of the second type;
determining the predicted gesture based on an output from the machine learning model; and
performing, in response to determining the predicted gesture, a predetermined action on the device.
2. The method of claim 1 , wherein the first sensor of the device comprises a photoplethysmography (PPG) sensor.
3. The method of claim 2 , wherein the PPG sensor comprises at least one of an infrared light source or a color light source.
4. The method of claim 2 , wherein the first sensor output indicates a change in blood flow.
5. The method of claim 1 , wherein the second sensor of the device comprises at least one of an accelerometer or a microphone.
6. The method of claim 1 , wherein at least one of receiving the first sensor output or receiving the second sensor output is based on a determination that the device is in a gesture detection mode.
7. The method of claim 1 , the machine learning model having been trained across multiple different users.
8. The method of claim 1 , wherein the predicted gesture comprises at least one of a finger-based gesture, or a wrist-based gesture.
9. The method of claim 8 , wherein the finger-based gesture comprises at least one of a finger pinch gesture or a fist-clinch gesture.
10. The method of claim 1 , wherein the predetermined action corresponds to changing a user interface on the device.
11. A device, comprising:
a first sensor;
a second sensor;
at least one processor; and
a memory including instructions that, when executed by the at least one processor, cause the at least one processor to:
receive, from the first sensor, first sensor output of a first type;
receive, from the second sensor, second sensor output of a second type that differs from the first type;
provide the first sensor output and the second sensor output as inputs to a machine learning model, the machine learning model having been trained to output a predicted gesture based on sensor output of the first type and sensor output of the second type;
determine the predicted gesture based on an output from the machine learning model; and
perform, in response to determining the predicted gesture, a predetermined action on the device.
12. The device of claim 11 , wherein the first sensor comprises a photoplethysmography (PPG) sensor.
13. The device of claim 12 , wherein the PPG sensor comprises at least one of an infrared light source or a color light source.
14. The device of claim 12 , wherein the first sensor output indicates a change in blood flow.
15. The device of claim 11 , wherein the second sensor the device comprises at least one of an accelerometer or a microphone.
16. The device of claim 11 , wherein at least one of receiving the first sensor output or receiving the second sensor output is based on a determination that the device is in a gesture detection mode.
17. The device of claim 11 , the machine learning model having been trained across a general population of users.
18. The device of claim 11 , wherein the predicted gesture comprises at least one of a finger-based gesture, or a wrist-based gesture.
19. The device of claim 18 , wherein the finger-based gesture comprises at least one of a finger pinch gesture or a fist-clinch gesture.
20. A computer program product comprising code, stored in a non-transitory computer-readable storage medium, the code comprising:
code to receive, from a first sensor of a device, first sensor output of a first type, the first sensor comprising a bio-signal sensor;
code to receive, from a second sensor of the device, second sensor output of a second type that differs from the first type;
code to provide the first sensor output and the second sensor output as inputs to a machine learning model, the machine learning model having been trained to output a predicted gesture based on sensor output of the first type and sensor output of the second type;
code to determine the predicted gesture based on an output from the machine learning model; and
code to perform, in response to determining the predicted gesture, a predetermined action on the device.
21. The computer program product of claim 20 , the code further comprising:
code to perform a gesture registration processes for a user of the device; and
code to tune the machine learning model for the user based on the gesture registration process.