Machine-learning based gesture recognition using multiple sensors
A device implementing a system for machine-learning based gesture recognition includes at least one processor configured to, receive, from a first sensor of the device, first sensor output of a first type, and receive, from a second sensor of the device, second sensor output of a second type that differs from the first type. The at least one processor is further configured to provide the first sensor output and the second sensor output as inputs to a machine learning model, the machine learning model having been trained to output a predicted gesture based on sensor output of the first type and sensor output of the second type. The at least one processor is further configured to determine the predicted gesture based on an output from the machine learning model, and to perform, in response to determining the predicted gesture, a predetermined action on the device.
1. A method, comprising:
receiving sensor data from a sensor of a device;
providing the sensor data as input to a machine learning model, the machine learning model having been trained to output, for each of a plurality of prediction windows that each include a portion of the sensor data, a gesture label and a no-gesture label;
determining a predicted gesture based on an output from the machine learning model; and
performing, in response to determining the predicted gesture, a predetermined action on the device.
2. The method of claim 1 , wherein, for each of the plurality of prediction windows, the gesture label and the no-gesture label each comprise a smoothed gesture label having a respective value between a maximum value for the smoothed gesture label and a minimum value for the smoothed gesture label.
3. The method of claim 2 , wherein the respective value of the smoothed gesture label is a value on a linear curve, a sigmoid curve, or an exponential curve between the minimum value and the maximum value of the smoothed gesture label.
4. The method of claim 1 , wherein, for each of the plurality of prediction windows, the gesture label comprises a probability that a gesture is occurring in that prediction window, and the no-gesture label comprises a probability that no gesture is occurring in that window.
5. The method of claim 4 , wherein, for each of the plurality of prediction windows, the output of the machine learning model comprises the gesture label, the no-gesture label, and a prediction of an amount of the predicted gesture that overlaps with a current portion of the sensor data.
6. The method of claim 4 , wherein determining the predicted gesture based on the output from the machine learning model comprises determining the predicted gesture based on a gesture predicted for the portion of the sensor data corresponding to the prediction window having a highest gesture label and a lowest no-gesture label.
7. The method of claim 4 , wherein determining the predicted gesture based on the output from the machine learning model comprises:
determining, based on the gesture label and the no-gesture label for each of the plurality of prediction windows, a start time and an end time of the predicted gesture; and
performing a gesture prediction operation for a set of the sensor data between the start time and the end time.
8. The method of claim 1 , wherein the predicted gesture comprises one of: a finger-based gesture or a wrist-based gesture.
9. The method of claim 1 , wherein performing the predetermined action on the device comprises providing a user input based on the predicted gesture to an application running at the device, the application including one of: a media player application, a workout application, an augmented reality application, a virtual reality application, or a health related application.
10. A device, comprising:
a sensor; and
at least one processor configured to:
receive sensor data from the sensor;
provide the sensor data as input to a machine learning model, the machine learning model having been trained to output, for each of a plurality of prediction windows that each include a portion of the sensor data, a gesture label and a no-gesture label;
determine a predicted gesture based on an output from the machine learning model; and
perform, in response to determining the predicted gesture, a predetermined action on the device.
11. The device of claim 10 , wherein, for each of the plurality of prediction windows, the gesture label and the no-gesture label each comprise a smoothed gesture label having a respective value between a maximum value for the smoothed gesture label and a minimum value for the smoothed gesture label.
12. The device of claim 11 , wherein the respective value of the smoothed gesture label is a value on a linear curve, a sigmoid curve, or an exponential curve between the minimum value and the maximum value of the smoothed gesture label.
13. The device of claim 10 , wherein, for each of the plurality of prediction windows, the gesture label comprises a probability that a gesture is occurring in that prediction window, and the no-gesture label comprises a probability that no gesture is occurring in that window.
14. The device of claim 13 , wherein, for each of the plurality of prediction windows, the output of the machine learning model comprises the gesture label, the no-gesture label, and a prediction of an amount of the predicted gesture that overlaps with a current portion of the sensor data.
15. The device of claim 13 , wherein the at least one processor is configured to determine the predicted gesture based on the output from the machine learning model based on a gesture predicted for the portion of the sensor data corresponding to the prediction window having a highest gesture label and a lowest no-gesture label.
16. The device of claim 15 wherein the at least one processor is configured to determine the predicted gesture by:
determining, based on the gesture label and the no-gesture label for each of the plurality of prediction windows, a start time and an end time of the predicted gesture; and
performing a gesture prediction operation for a set of the sensor data between the start time and the end time.
17. A non-transitory computer readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving sensor data from a sensor of a device;
providing the sensor data as input to a machine learning model, the machine learning model having been trained to output, for each of a plurality of prediction windows that each include a portion of the sensor data, a gesture label and a no-gesture label;
determining a predicted gesture based on an output from the machine learning model; and
performing, in response to determining the predicted gesture, a predetermined action on the device.
18. The non-transitory computer readable medium of claim 17 , wherein, for each of the plurality of prediction windows, the gesture label and the no-gesture label each comprise a smoothed gesture label having a respective value between a maximum value for the smoothed gesture label and a minimum value for the smoothed gesture label.
19. The non-transitory computer readable medium of claim 18 , wherein the respective value of the smoothed gesture label is a value on a linear curve, a sigmoid curve, or an exponential curve between the minimum value and the maximum value of the smoothed gesture label.
20. The non-transitory computer readable medium of claim 17 , wherein, for each of the plurality of prediction windows, the gesture label comprises a probability that a gesture is occurring in that prediction window, and the no-gesture label comprises a probability that no gesture is occurring in that window.