Human-computer interface for multimodal conversational interactions in mobile contexts
A physical input from a user is detected at a sensor included in an annular member having a central opening. Audio from the user is received at a microphone included in the annular member. Feedback is provided to the user by a feedback indicator included in the annular member. The received audio is provided to a computing device from a communication interface included in the annular member.
1 . A system, comprising:
an annular member having a central opening;
a sensor included in the annular member and configured to detect a physical input from a user;
an inertial measurement unit (IMU) configured to output a signal indicative of an orientation and/or motion of the annular member in three-dimensional space;
a processor configured to:
receive an output from the sensor and the output from the IMU;
determine, based in part of the output of the IMU that the annular member is near a mouth of the user; and
in response to receiving an output from the sensor and the output from the IMU, determine a type of input primitive associated with the physical input based on the output from the sensor being received within a threshold of the output from the IMU;
a microphone included in the annular member and configured to receive audio from the user;
a communication interface included in the annular member and configured to provide the received audio to a computing device; and
a feedback indicator included in the annular member and configured to provide feedback to the user.
2 . The system of claim 1 , wherein the processor is further configured to:
determine that the physical input from the user is detected at the sensor;
provide, via the feedback indicator, feedback to the user indicating that the physical input was detected;
receive, via the microphone, the audio;
determine whether the physical input is still detected at the sensor; and
in response to a determination that the physical input is not detected at the sensor, provide the received audio, via the communication interface, to the computing device.
3 . The system of claim 2 , wherein the processor is further configured to:
provide, via the feedback indicator, feedback to the user indicating that the user audio has been received.
4 . The system of claim 3 , wherein the processor is configured to:
receive, via the communication interface, a response from the computing device at the communication interface; and
provide the response to the user.
5 . The system of claim 4 , wherein the response is received directly from the computing device.
6 . The system of claim 4 , wherein the response is received from the computing device via a user device.
7 . The system of claim 4 , wherein the processor is further configured to:
while the response is provided to the user, determine whether the user has interrupted the response.
8 . The system of claim 7 , wherein the processor is further configured to determine whether the response is finished in response to a determination that the user did not interrupt the response.
9 . The system of claim 8 , wherein the processor is further configured to provide the response to the user in response to a determination that the response is not finished.
10 . The system of claim 7 , wherein in response to the response being interrupted, the microphone is configured to receive new user audio from the user.
11 . The system of claim 1 , wherein the computing device includes an artificial intelligence (AI) service which includes a large language model (LLM).
12 . The system of claim 11 , wherein the AI service is configured to receive and respond to multimodal inputs from the user.
13 . The system of claim 1 , wherein the annular member is worn on a first finger and the sensor is positioned and configured to receive the physical input from the user via a second finger.
14 . The system of claim 1 , wherein the feedback indicator includes a motor.
15 . The system of claim 1 , wherein the feedback indicator provides haptic feedback to the user.
16 . The system of claim 2 , further comprising a second sensor, wherein the processor is further configured to:
detect, via the second sensor, a motion performed by the user; and
in response to the detection of the motion performed by the user, execute a command.
17 . A method, comprising:
detecting, by a sensor included in an annular member, a physical input from a user;
outputting, by an inertial measurement unit (IMU) included in the annular member, a signal indicative of an orientation and/or motion of the annular member in three-dimensional space;
receiving, by a processor included in the annular member, an output from the sensor and the output from the IMU;
determining, by the processor, based in part on the output of the IMU that the annular member is near a mouth of the user;
in response to receiving an output from the sensor and the output from the IMU, determining a type of input primitive associated with the physical input based on the output from the sensor being received within a threshold of the output from the IMU;
receiving, by a microphone included in the annular member, audio from the user;
providing, by a feedback indicator included in the annular member, feedback to the user; and
providing, by a communication interface included in the annular member, the received audio to a computing device.
18 . The method of claim 17 , further comprising:
determining whether the physical input is still detected by the sensor included in an annular member;
providing, via the feedback indicator, feedback to the user indicating that a physical input has been detected;
receiving, via the microphone, the audio;
determining whether the physical input is still detected at the sensor; and
in response to determining that the physical input is not detected, providing, via the communication interface included in the annular member, the received audio to a computing device.
19 . A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:
detecting, by a sensor included in an annular member, a physical input from a user;
outputting, by an inertial measurement unit (IMU) included in the annular member, a signal indicative of an orientation and/or motion of the annular member in three-dimensional space;
receiving, by a processor included in the annular member, an output from the sensor and the output from the IMU;
determining, by the processor, based in part on the output of the IMU that the annular member is near a mouth of the user;
in response to receiving an output from the sensor and the output from the IMU, determining a type of input primitive associated with the physical input based on the output from the sensor being received within a threshold of the output from the IMU;
providing, by a feedback indicator included in the annular member, feedback to the user indicating that a command associated with the physical input was initiated; and
providing, by a communication interface included in the annular member, the received audio to a computing device.