Inertial sensing of tongue gestures
This document relates to employing tongue gestures to control a computing device, and training machine learning models to detect tongue gestures. One example relates to a method or technique that can include receiving one or more motion signals from an inertial sensor. The method or technique can also include detecting a tongue gesture based at least on the one or more motion signals, and outputting the tongue gesture.
1. A method comprising:
displaying a plurality of items;
detecting scanning over the plurality of items based on user input;
obtaining one or more motion signals from an inertial sensor, the one or more motion signals obtained from the inertial sensor reflecting movement of a tongue of a user while the user input is directed to a particular item;
inputting the one or more motion signals to a machine learning model that has been trained to detect one or more tongue gestures;
receiving, from the machine learning model, an indication that a particular tongue gesture is detected from the one or more motion signals; and
selecting the particular item in response to receiving the indication that the particular tongue gesture has been detected by the machine learning model.
2. The method of claim 1 , wherein the one or more motion signals are provided by an inertial measurement unit.
3. The method of claim 1 , wherein the plurality of items comprise letters.
4. The method of claim 3 , wherein the letters are displayed on a virtual keyboard.
5. The method of claim 4 , wherein selecting the particular item comprises entering a particular letter from the virtual keyboard as text.
6. The method of claim 5 , further comprising backspacing over the particular letter in response to detecting another tongue gesture from the one or more motion sensor signals with the machine learning model.
7. The method of claim 6 , the tongue gesture comprising a teeth tap and the another tongue gesture comprising a cheek tap.
8. The method of claim 5 , the user input comprising eye gaze input that scans over the letters on the virtual keyboard and is directed to the particular letter when the tongue gesture is detected.
9. The method of claim 8 , further comprising:
visually distinguishing currently-targeted letters on the virtual keyboard as the eye gaze input scans over the virtual keyboard.
10. The method of claim 9 , the currently-targeted letters being bolded or enlarged when currently targeted by the eye gaze input.
11. A system comprising:
an inertial measurement unit configured to provide motion signals reflecting movement of a tongue of a user;
a processor; and
a computer-readable storage medium storing instructions which, when executed by the processor, cause the system to:
output a plurality of items for display;
detect scanning over the plurality of items based on user input;
input the motion signals to a machine learning model that has been trained to detect one or more tongue gestures;
receive, from the machine learning model, an indication that a particular tongue gesture is detected from the motion signals; and
select the particular item in response to receiving the indication that the particular tongue gesture has been detected by the machine learning model.
12. The system of claim 11 , wherein the instructions, when executed by the processor, cause the system to:
visually distinguish currently-targeted items when scanned over by the user input.
13. The system of claim 12 , wherein the instructions, when executed by the processor, cause the system to:
at least one of bold or enlarge the currently-targeted items when scanned over by the user input.
14. The system of claim 13 , further comprising an eye tracking sensor, the user input comprising eye gaze input detected by the eye tracking sensor.
15. The system of claim 14 , further comprising a display configured to display the plurality of items.
16. The system of claim 15 , the inertial measurement unit, the processor, the computer-readable storage medium, the eye tracking sensor, and the display being integrated into a virtual or augmented reality headset.
17. The system of claim 16 , wherein the inertial measurement unit comprises an accelerometer, a gyroscope, and a magnetometer.
18. A computer-readable storage medium storing instructions which, when executed by a processor, cause the processor to perform acts comprising:
outputting a plurality of items for display;
detecting scanning over the plurality of items based on user input;
obtaining one or more motion signals from an inertial sensor, the one or more motion signals obtained from the inertial sensor reflecting movement of a tongue of a user while the user input is directed to a particular item;
inputting the one or more motion signals to a machine learning model that has been trained to detect one or more tongue gestures;
receiving, from the machine learning model, an indication that a particular tongue gesture is detected from the one or more motion signals; and
selecting the particular item in response to receiving the indication that the particular tongue gesture has been detected by the machine learning model.
19. The computer-readable storage medium of claim 18 , the machine learning model being a random forest.
20. The computer-readable storage medium of claim 18 , the acts further comprising:
obtaining one or more other signals from another sensor; and
inputting the one or more other signals to the machine learning model, wherein the indication that the particular tongue gesture has been detected is also based on the one or more other signals obtained from the another sensor.