Multi-modal gesture sensing on wearable devices
Techniques include using, in addition to IMU signals, audio captured by one or more microphones on a wearable device to determine whether a gesture has been made, and if so, which gesture has been made. The IMUs provide a first electronic signal to processing circuitry on the wearable device in response to detected motion. The one or more microphones provide a second electronic signal to the processing circuitry in response to detected audio resulting from a gesture. The processing circuitry determines whether a predefined gesture has been made based at least on the first electronic signal and the second electronic signal. If the processing circuitry determines that the predefined gesture was made, the processing circuitry performs an operation on the wearable device corresponding to the predefined gesture.
1 . A method comprising:
receiving a first electronic signal with audio data, ultrasound data, or radio frequency data from a sensor of a wearable device;
receiving a second electronic signal including motion data from a motion sensor of the wearable device;
generating feature data from the first electronic signal and the second electronic signal, the feature data being configured to include values from the audio data, ultrasound data, radio frequency data, and motion data;
generating an indicator of a gesture that is predefined based on the feature data, the indicator including information that indicates whether a user has made the gesture; and
in response to the indicator indicating that the user has made the gesture, causing an interaction with a model on the wearable device.
2 . The method of claim 1 , wherein the first electronic signal includes the audio data from one or more microphones on the wearable device.
3 . The method of claim 2 , further comprising:
receiving a third electronic signal with the ultrasound data from an ultrasonic frequency sensor on the wearable device; and
generating the indicator using the first electronic signal, the second electronic signal, and the third electronic signal.
4 . The method of claim 2 , further comprising:
receiving a third electronic signal with the radio frequency data from a radio frequency sensor; and
generating the indicator using the first electronic signal, the second electronic signal, and the third electronic signal.
5 . The method of claim 1 , wherein generating the indicator includes:
inputting the first electronic signal and the second electronic signal into a machine-learning (ML) model, wherein the indicator includes an identifier of the gesture.
6 . The method of claim 5 , further comprising:
synchronizing samples of the first electronic signal and the second electronic signal in time to produce synchronized samples of the first electronic signal and the second electronic signal;
combining the synchronized samples of the first electronic signal and the second electronic signal to produce the feature data; and
inputting the feature data into the ML model.
7 . The method of claim 6 , wherein combining the synchronized samples of the first electronic signal and the second electronic signal includes:
concatenating the synchronized samples of the first electronic signal and the second electronic signal.
8 . The method of claim 6 , further comprising:
applying a signal filter to the synchronized samples; and
converting the synchronized samples to a frequency domain using one or more transforms.
9 . The method of claim 1 , wherein the first electronic signal includes samples at a first sampling rate, and the second electronic signal includes samples at a second sampling rate, the second sampling rate being different than the first sampling rate.
10 . The method of claim 1 , wherein the model is a language model and causing the interaction includes initiating an action with respect to the language model.
11 . A wearable device comprising:
at least one processor; and
a non-transitory computer-readable medium storing executable instructions that cause the at least one processor to execute operations, the operations comprising:
receiving a first electronic signal with audio data, ultrasound data, or radio frequency data from a sensor of the wearable device;
receiving a second electronic signal with motion data from a motion sensor of the wearable device;
generating feature data from the first electronic signal and the second electronic signal, the feature data being configured to include values from the audio data, ultrasound data, radio frequency data, and motion data;
generating an indicator of a predefined gesture based on the feature data, the indicator including information that indicates whether a user has made the predefined gesture; and
in response to the indicator indicating that the user has made the predefined gesture, causing an interaction with a model on the wearable device.
12 . The wearable device of claim 11 , wherein the first electronic signal includes the ultrasound data from an ultrasonic frequency sensor on the wearable device.
13 . The wearable device of claim 12 , wherein the operations further comprise:
receiving a third electronic signal with the audio data from one or more microphones on the wearable device; and
generating the indicator using the first electronic signal, the second electronic signal, and the third electronic signal.
14 . The wearable device of claim 12 , wherein the operations further comprise:
receiving a third electronic signal with the radio frequency data from a radio frequency sensor; and
generating the indicator using the first electronic signal, the second electronic signal, and the third electronic signal.
15 . The wearable device of claim 11 , wherein the operations further comprise:
inputting the feature data into a machine-learning (ML) model, the indicator including an identifier of the predefined gesture.
16 . The wearable device of claim 15 , wherein the operations further comprise:
applying one or more signal filters to the first electronic signal and the second electronic signal;
applying one or more frequency transforms to the first electronic signal and the second electronic signal; and
combining the first electronic signal and the second electronic signal to form the feature data.
17 . A non-transitory computer-readable medium storing executable instructions that when executed by at least one processor cause the at least one processor to execute operations, the operations comprising:
receiving a first electronic signal with audio data, ultrasound data, or radio frequency data from a sensor of a wearable device;
receiving a second electronic signal with motion data from a motion sensor of the wearable device;
generating feature data from the first electronic signal and the second electronic signal, the feature data being configured to include values from the audio data, ultrasound data, radio frequency data, and motion data;
generating an indicator of a predefined gesture based on the feature data, the indicator including information that indicates whether a user has made the predefined gesture; and
in response to the indicator indicating that the user has made the predefined gesture, causing an interaction with a model on the wearable device.
18 . The non-transitory computer-readable medium of claim 17 , wherein the first electronic signal includes the radio frequency data from an ultrasonic frequency sensor on the wearable device.
19 . The non-transitory computer-readable medium of claim 18 , wherein the operations further comprise:
receiving a third electronic signal with the audio data from one or more microphones on the wearable device; and
generating the indicator using the first electronic signal, the second electronic signal, and the third electronic signal.
20 . The non-transitory computer-readable medium of claim 18 , wherein the operations further comprise:
receiving a third electronic signal with the ultrasound data from the ultrasonic frequency sensor; and
generating the indicator using the first electronic signal, the second electronic signal, and the third electronic signal.
21 . The method of claim 1 , wherein the interaction with the model includes at least one of invoking the model, transmitting a prompt to the model, confirming a response from the model, or rejecting the response from the model.
22 . The method of claim 1 , wherein the first electronic signal includes radio frequency data and the indicator of the gesture is generated based on a disruption in the radio frequency data.