Active Listening for Assistant Systems
In one embodiment, a method includes receiving a first user input comprising a wake word associated with an assistant xbot from a first client system, setting the assistant xbot into a listening mode, wherein a continuous non-visual feedback is provided via the first client system while the assistant xbot is in the listening mode, receiving a second user input comprising a user utterance from the first client system while the assistant xbot is in the listening mode, determining the second user input has ended based on a completion of the user utterance, and setting the assistant xbot into an inactive mode, wherein the non-visual feedback is discontinued via the first client system while the assistant xbot is in the inactive mode.
1 . (canceled)
2 . A method of operating a head-mounted device, the method comprising:
receiving, at a sensor of the head-mounted device, a first input via a visual modality, wherein the first input comprises an indication of a gaze of a user of the head-mounted device;
processing, by the head-mounted device, the first input to detect a wake instruction, wherein the wake instruction is detected based at least in part on the gaze of the user;
in response to detecting the wake instruction, activating an assistant system;
receiving, at the head-mounted device, a second input via a second modality, wherein the second input comprises a user request;
communicating the user request to the activated assistant system;
receiving, from the activated assistant system, a response to the user request; and
presenting, via an I/O device of the head-mounted device, the response to the user request.
3 . The method of claim 2 , wherein the second modality is different from the visual modality.
4 . The method of claim 3 , wherein the second modality is a voice modality, the second input comprises speech of the user, and the user request is determined based at least in part on the speech of the user.
5 . The method of claim 2 , wherein the assistant system is installed on a companion device, and the head-mounted device communicates with the companion device via a direct wireless connection between the head-mounted device and the companion device.
6 . The method of claim 5 , wherein the assistant system installed on the companion device communicates with a remote server, and the user request is processed, at least in part, at the remote server.
7 . The method of claim 2 , further comprising:
in response to detecting the wake instruction, placing the assistant system in a listening mode, and
providing an indication, via the head-mounted device, that the assistant system was placed in the listening mode.
8 . The method of claim 7 , wherein the indication persists, to provide continuous feedback, while the assistant system is in the listening mode.
9 . The method of claim 5 ,
wherein, when the assistant system is not activated, the assistant system executes as a background application of the companion device.
10 . The method of claim 2 ,
wherein the presented response comprises at least one of: an audio output or a visual output.
11 . A non-transitory computer-readable medium comprising storing instructions for operating a head-mounted device that, when executed by one or more processors of a computing system, cause the computing system to:
receive, at a sensor of the head-mounted device, a first input via a visual modality, wherein the first input comprises an indication of a gaze of a user of the head-mounted device;
process, by the head-mounted device, the first input to detect a wake instruction, wherein the wake instruction is detected based at least in part on the gaze of the user;
in response to detecting the wake instruction, activate an assistant system;
receive, at the head-mounted device, a second input via a second modality, wherein the second input comprises a user request;
communicate the user request to the activated assistant system;
receive, from the activated assistant system, a response to the user request; and
present, via an I/O device of the head-mounted device, the response to the user request.
12 . The non-transitory computer-readable medium of claim 11 , wherein the second modality is different from the visual modality.
13 . The non-transitory computer-readable medium of claim 12 , wherein the second modality is a voice modality, the second input comprises speech of the user, and the user request is determined based at least in part on the speech of the user.
14 . The non-transitory computer-readable medium of claim 11 , wherein the assistant system is installed on a companion device, and the head-mounted device communicates with the companion device via a direct wireless connection between the head-mounted device and the companion device.
15 . The non-transitory computer-readable medium of claim 14 , wherein the assistant system installed on the companion device communicates with a remote server, and the user request is processed, at least in part, at the remote server.
16 . The non-transitory computer-readable medium of claim 11 , wherein the instructions, when executed by one or more processors of the computing system, further cause the computing system to:
in response to detecting the wake instruction, place the assistant system in a listening mode, and
provide an indication, via the head-mounted device, that the assistant system was placed in the listening mode.
17 . The non-transitory computer-readable medium of claim 16 , wherein the indication persists, to provide continuous feedback, while the assistant system is in the listening mode.
18 . The non-transitory computer-readable medium of claim 14 ,
wherein, when the assistant system is not activated, the assistant system executes as a background application of the companion device.
19 . The non-transitory computer-readable medium of claim 11 ,
wherein the presented response comprises at least one of: an audio output or a visual output.
20 . A system, comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions for operating a head-mounted device that, when executed by at least one of the one or more processors, cause the system to:
receive, at a sensor of the head-mounted device, a first input via a visual modality, wherein the first input comprises an indication of a gaze of a user of the head-mounted device;
process, by the head-mounted device, the first input to detect a wake instruction, wherein the wake instruction is detected based at least in part on the gaze of the user;
in response to detecting the wake instruction, activate an assistant system;
receive, at the head-mounted device, a second input via a second modality, wherein the second input comprises a user request;
communicate the user request to the activated assistant system;
receive, from the activated assistant system, a response to the user request; and
present, via an I/O device of the head-mounted device, the response to the user request.
21 . The system of claim 20 , wherein the second modality is a voice modality, the second input comprises speech of the user, and the user request is determined based at least in part on the speech of the user.