Inertial sensing of tongue gestures
This document relates to employing tongue gestures to control a computing device, and training machine learning models to detect tongue gestures. One example relates to a method or technique that can include receiving one or more motion signals from an inertial sensor. The method or technique can also include detecting a tongue gesture based at least on the one or more motion signals, and outputting the tongue gesture.
1. A method comprising:
receiving one or more motion signals from an inertial sensor, the one or more motion signals received from the inertial sensor reflecting movement of a tongue of a user;
inputting the one or more motion signals to a machine learning model that has been trained to detect one or more tongue gestures;
receiving, from the machine learning model, an indication that a particular tongue gesture is detected from the one or more motion signals; and
outputting the particular tongue gesture to control an application.
2. The method of claim 1 , wherein the particular tongue gesture comprises one or more taps on one or more front upper teeth by the tongue of the user.
3. The method of claim 1 , wherein the particular tongue gesture comprises lateral movement of the tongue of the user, the lateral movement including a tap of a left or right cheek of the user.
4. The method of claim 1 , wherein the particular tongue gesture comprises a swing tongue sideways gesture, a mouth floor tongue gesture, a curl back tongue gesture, or a tongue bite gesture.
5. The method of claim 1 , wherein the inertial sensor is provided in a virtual or augmented reality headset, earbuds, headphones, or a cochlear implant.
6. The method of claim 1 , further comprising:
receiving one or more other signals from another sensor; and
inputting the one or more other signals to the machine learning model, wherein the indication that the particular tongue gesture has been detected is also based on the one or more other signals received from the another sensor.
7. A method comprising:
instructing a user to perform a particular tongue gesture;
measuring one or more motion signals from an inertial sensor while the user performs the particular tongue gesture;
training a machine learning model to detect the particular tongue gesture using the one or more motion signals; and
outputting the trained machine learning model.
8. The method of claim 7 , wherein the training is performed using supervised learning using the particular tongue gesture as a label for the one or more motion signals.
9. The method of claim 8 , wherein the inertial sensor comprises an accelerometer, a gyroscope, and a magnetometer.
10. The method of claim 9 , further comprising:
performing principal component analysis on a moving time window of the one or more motion signals to extract one or more principal components and employing the one or more principal components to perform the supervised learning.
11. The method of claim 10 , wherein the machine learning model comprises a random forest that includes multiple decision trees.
12. The method of claim 11 , further comprising:
providing moving windows of the principal components to individual decision trees of the random forest;
determining a majority vote of the individual decision trees; and
updating parameters of the random forest based at least on whether the majority vote matches the particular tongue gesture that the user was instructed to perform.
13. The method of claim 8 , further comprising:
performing the training using training data for a plurality of users; and
performing individualized tuning of the trained machine learning model to at least two other users responsive to performance of the particular tongue gesture by the at least two other users.
14. A system comprising:
an inertial measurement unit configured to provide motion signals reflecting movement of a tongue of a user;
a processor; and
a computer-readable storage medium storing instructions which, when executed by the processor, cause the system to:
input the motion signals reflecting the movement of the tongue of the user to a machine learning model that has been trained to detect one or more tongue gestures;
receive, from the machine learning model, an indication that a particular tongue gesture is detected from the motion signals; and
output the particular tongue gesture to control an application.
15. The system of claim 14 , provided in a virtual or augmented reality headset.
16. The system of claim 15 , wherein the inertial measurement unit is provided within a face gasket of the virtual or augmented reality headset.
17. The system of claim 14 , wherein the instructions, when executed by the processor, cause the system to:
display a plurality of items;
scan over the plurality of items in response to eye gaze tracked by an eye tracking component; and
in response to the particular tongue gesture being detected by the machine learning model while the eye gaze is directed to a particular item, select the particular item.
18. The system of claim 17 wherein the instructions, when executed by the processor, cause the system to:
deselect the particular item responsive to another tongue gesture.
19. The system of claim 14 , wherein the inertial measurement unit is provided in headphones or earbuds.
20. The system of claim 14 , the application comprising an audio-only application that does not use a visual display.
21. The system of claim 14 , wherein the machine learning model comprises a random forest that includes multiple decision trees.