Devices and methods for digital assistant
A digital assistant may be used with a computer-generated environment. In some embodiments, a digital assistant is invoked and/or terminated via user input. In some embodiments, the digital assistant is displaying in a computer-generated environment. In some embodiments, the digital assistant may be used to interact with objects in a computer-generated environment. Devices, methods, and graphical user interfaces for a digital assistant provide an improved user experience for computer-generated environments.
1. A method comprising:
at an electronic device in communication with a display and one or more input devices comprising at least one hand-tracking sensor:
displaying, via the display, a three-dimensional environment;
while displaying the three-dimensional environment, detecting, via the at least one hand-tracking sensor, a first input including a gesture by a hand; and
in response to the first input:
in accordance with a determination that the first input satisfies first criteria, the first criteria including a first criterion that is satisfied when the hand is oriented in a specified direction relative to the electronic device or within a threshold of the specified direction relative to the electronic device, when a dorsal aspect of the hand is facing the electronic device and including a second criterion that is satisfied when a gaze is detected at or within a threshold distance of the hand or a representation of the hand in the three-dimensional environment, activating a digital assistant and displaying a visual representation of the digital assistant in the three-dimensional environment, wherein;
the digital assistant interprets natural language input and performs one or more actions based on the natural language input; and
in accordance with a determination that the digital assistant is activated:
obtaining audio data; and
changing the visual representation of the digital assistant based on the audio data while obtaining the audio data; and
in accordance with a determination that the first input fails to satisfy the first criteria, forgoing activating the digital assistant.
2. The method of claim 1 , further comprising: while the digital assistant is activated and while displaying the three-dimensional environment, detecting, via the one or more input devices, a second input; and in response to the second input: in accordance with a determination that the second input satisfies one or more second criteria, ceasing displaying of the visual representation of the digital assistant; and in accordance with a determination that the second input fails to satisfy the one or more second criteria, continuing displaying the visual representation of the digital assistant.
3. The method of claim 1 , wherein the visual representation of the digital assistant in the three-dimensional environment is anchored to a representation of the hand in the three-dimensional environment.
4. The method of claim 1 , wherein the first criteria include one or more of: a criterion that is satisfied when the hand corresponds to a predetermined hand; a criterion that is satisfied when the hand is in a predetermined pose; a criterion that is satisfied when the hand is orientated in a specified direction or within a threshold of the specified direction; or a criterion that is satisfied when the hand is within a field of view of a sensor of the electronic device.
5. The method of claim 1 , wherein the first criteria include a criterion that is satisfied when a gaze is detected at or within a threshold distance of the hand or a representation of the hand in the three-dimensional environment.
6. The method of claim 1 , wherein the first criteria include a criterion that is satisfied when a subset of the first criteria are satisfied for a threshold period of time.
7. The method of claim 1 , further comprising: while the digital assistant is activated and while displaying the three-dimensional environment, detecting, via the one or more input devices, audio data; and in response to the audio data: displaying a representation of the audio data in the three-dimensional environment.
8. An electronic device comprising: a display; one or more input devices comprising at least one hand-tracking sensor; one or more processors; memory; and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for: displaying, via the display, a three-dimensional environment; while displaying the three-dimensional environment, detecting, via the at least one hand-tracking sensor a first input including a gesture by a hand; and in response to the first input: in accordance with a determination that the first input satisfies first criteria, the first criteria including a criterion that is satisfied when the hand is oriented in a specified direction relative to the electronic device or within a threshold of the specified direction relative to the electronic device, when a dorsal aspect of the hand is facing the electronic device and a criterion that is satisfied when a gaze is detected at or within a threshold distance of the hand or a representation of the hand in the three-dimensional environment, activating a digital assistant and displaying a visual representation of the digital assistant in the three-dimensional environment, wherein:
the digital assistant interprets natural language input and performs one or more actions based on the natural language input; and
in accordance with a determination that a digital assistant is activated:
obtaining audio data; and
changing the visual representation of the digital assistant based on the audio data while obtaining the audio data; and
in accordance with a determination that the first input fails to satisfy the first criteria, forgoing activating the digital assistant.
9. A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device, cause the electronic device to perform a method comprising:
displaying, via a display, a three-dimensional environment; while displaying the three-dimensional environment, detect, via one or more input devices comprising at least one hand-tracking sensor, a first input including a gesture by a hand; and in response to the first input: in accordance with a determination that the first input satisfies first criteria, the first criteria including a criterion that is satisfied when the hand is oriented in a specified direction relative to the electronic device or within a threshold of the specified direction relative to the electronic device, when a dorsal aspect of the hand is facing the electronic device and a criterion that is satisfied when a gaze is detected at or within a threshold distance of the hand or a representation of the hand in the three-dimensional environment, activating a digital assistant and displaying a visual representation of the digital assistant in the three-dimensional environment, wherein:
the digital assistant interprets natural language input and performs one or more actions based on the natural language input; and
in accordance with a determination that a digital assistant is activated:
obtaining audio data; and
changing the visual representation of the digital assistant based on the audio data while obtaining the audio data; and
in accordance with a determination that the first input fails to satisfy the first criteria, forgoing activating the digital assistant.
10. The electronic device of claim 8 , the one or more programs further including instructions for: while the digital assistant is activated and while displaying the three-dimensional environment, detecting, via the one or more input devices, a second input; and in response to the second input: in accordance with a determination that the second input satisfies one or more second criteria, ceasing displaying of the visual representation of the digital assistant; and in accordance with a determination that the second input fails to satisfy the one or more second criteria, continuing displaying the visual representation of the digital assistant.
11. The electronic device of claim 8 , wherein the visual representation of the digital assistant in the three-dimensional environment is anchored to a representation of the hand in the three-dimensional environment.
12. The electronic device of claim 8 , wherein the digital assistant interprets natural language input and performs one or more actions based on the natural language input.
13. The electronic device of claim 8 , wherein the first criteria include one or more of: a criterion that is satisfied when the hand corresponds to a predetermined hand; a criterion that is satisfied when the hand is in a predetermined pose; a criterion that is satisfied when a gaze is detected at or within a threshold distance of the hand or a representation of the hand in the three-dimensional environment; or a criterion that is satisfied when the hand is within a field of view of a sensor of the electronic device.
14. The electronic device of claim 8 , the one or more programs further including instructions for: in response to the first input: in accordance with the determination that the first input satisfies the first criteria, obtaining audio data via an audio sensor; while the digital assistant is activated and while displaying the three-dimensional environment, detecting, via the one or more input devices, a second input; and in response to the second input: in accordance with a determination that the second input satisfies one or more second criteria, ceasing obtaining additional audio data; and in accordance with a determination that the second input fails to satisfy the one or more second criteria, continuing obtaining the audio data.
15. The electronic device of claim 14 , wherein the one or more second criteria include one or more of: a criterion that is satisfied when the hand is in a predetermined pose; a criterion that is satisfied when the hand is not oriented in a specified direction or within a threshold of the specified direction; a criterion that is satisfied when the audio data indicate a user ceases to speak for a threshold period of time; or a criterion that is satisfied when the audio data indicate a predefined audio command.
16. The electronic device of claim 14 , the one or more programs further including instructions for: in response to the first input: in accordance with the determination that the first input satisfies the first criteria, obtaining audio data via an audio sensor while the digital assistant is activated and while displaying the three-dimensional environment, detecting, via the one or more input devices, a second input; and in response to the second input: in accordance with a determination that the second input satisfies one or more second criteria, executing a command in accordance with the audio data; and in accordance with a determination that the second input fails to satisfy the one or more second criteria, continuing obtaining the audio data.
17. The electronic device of claim 16 , wherein the command comprises adding a new object or manipulating an existing object in the three-dimensional environment, the one or more programs further including instructions for: adding the new object or manipulating the existing object in accordance with executing the command.
18. The non-transitory computer readable storage medium of claim 9 , the method further comprising: while the digital assistant is activated and while displaying the three-dimensional environment, detecting, via the one or more input devices, a second input; and in response to the second input: in accordance with a determination that the second input satisfies one or more second criteria, deactivating the digital assistant; and in accordance with a determination that the second input fails to satisfy the one or more second criteria, continuing obtaining audio data.
19. The non-transitory computer readable storage medium of claim 9 , wherein the first criteria include one or more of: a criterion that is satisfied when the hand corresponds to a predetermined hand; a criterion that is satisfied when the hand is in a predetermined pose; a criterion that is satisfied when the hand is orientated in a specified direction or within a threshold of the specified direction; a criterion that is satisfied when the hand is within a field of view of a sensor of the electronic device; or a criterion that is satisfied when a subset of the first criteria are satisfied for a threshold period of time.
20. The non-transitory computer readable storage medium of claim 9 , wherein the digital assistant interprets natural language input and performs one or more actions based on the natural language input.
21. The method of claim 1 , wherein the first criteria include a criterion that is satisfied when the gesture by the hand includes a predetermined pose, wherein the predetermined pose corresponds to the hand in a fist.