Systems and methods for artificial intelligence-based virtual and augmented reality
Examples of the disclosure describe systems and methods for generating and displaying a virtual companion. In an example method, a first input from an environment of a user is received at a first time via a first sensor. An occurrence of an event in the environment is determined based on the first input. A second input from the user is received via a second sensor, and an emotional reaction of the user is identified based on the second input. An association is determined between the emotional reaction and the event. A view of the environment is presented at a second time later than the first time via a display. A stimulus is presented at the second time via a virtual companion displayed via the display, wherein the stimulus is determined based on the determined association between the emotional reaction and the event.
1. A method comprising:
receiving, at a first time, via a first sensor of a first device, a first input from an environment of a user;
receiving user data via a second device in communication with the first device via a network;
determining, based on the first input and the user data, an occurrence of an event in the environment;
receiving, via a second sensor of the first device, a second input from the user;
identifying, based on the second input, an emotional reaction of the user;
receiving, via the second sensor at a second time later than the first time, a third input from the user;
in accordance with a determination that the third input comprises a request for information about the event:
constructing a query based on the third input, wherein the query is associated with the event and further associated with the requested information; and
presenting a response to the query via a display; and
in accordance with a determination that the third input does not comprise a request for information about the event, forgoing constructing the query.
2. The method of claim 1 , wherein the first input comprises at least one of an image of a physical object and an audio signal.
3. The method of claim 1 , wherein the second input comprises speech of the user, and wherein said identifying the emotional reaction comprises determining a content of the speech.
4. The method of claim 1 , wherein the second input comprises an eye movement of the user, and wherein said identifying the emotional reaction comprises determining a gaze direction for the user based on the eye movement.
5. The method of claim 1 , wherein the second input comprises a field of view of the user, and wherein said identifying the emotional reaction comprises identifying at least one object within the field of view.
6. The method of claim 1 , further comprising determining an intensity of the emotional reaction, wherein the response to the query is determined further based on the intensity.
7. The method of claim 1 , wherein the display comprises a display of a wearable head device.
8. The method of claim 1 , wherein the third input is associated with an interaction between the user and a digital assistant.
9. The method of claim 1 , wherein the response is determined based on an association between the emotional reaction and the event.
10. The method of claim 9 , wherein the association between the emotional reaction and the event is determined based on one or more of the first input, the user data, and the second input.
11. The method of claim 9 , wherein the association between the emotional reaction and the event comprises at least one of a temporal association and a spatial association.
12. The method of claim 9 , wherein the event comprises a first event, the method further comprising storing the association between the emotional reaction and the first event in a memory graph, wherein the memory graph comprises an association between the first event and a second event.
13. The method of claim 1 , wherein the first device comprises a head-wearable device and the second device comprises a remote resource.
14. The method of claim 1 , wherein the determination that the third input comprises the request for the information about the event comprises a determination that the third input indicates an attempt by the user to recall the information about the event.
15. A system comprising:
a first sensor of a first device;
a second sensor of the first device;
a display; and
one or more processors configured to perform a method comprising:
receiving, at a first time, via the first sensor, a first input from an environment of a user;
receiving user data via a second device in communication with the first device via a network;
determining, based on the first input and the user data, an occurrence of an event in the environment;
receiving, via the second sensor, a second input from the user;
identifying, based on the second input, an emotional reaction of the user;
receiving, via the second sensor at a second time later than the first time, a third input from the user;
in accordance with a determination that the third input comprises a request for information about the event:
constructing a query based on the third input, wherein the query is associated with the event and further associated with the requested information; and
presenting a response to the query via the display; and
in accordance with a determination that the third input does not comprise a request for information about the event, forgoing constructing the query.
16. The system of claim 15 , wherein the second input comprises speech of the user, and wherein said identifying the emotional reaction comprises determining a content of the speech.
17. The system of claim 15 , the method further comprising determining an intensity of the emotional reaction, wherein the response to the query is determined further based on the intensity.
18. The system of claim 15 , wherein the display comprises a display of a wearable head device.
19. The system of claim 15 , wherein the third input is associated with an interaction between the user and a digital assistant.
20. The system of claim 15 , wherein the response is determined based on an association between the emotional reaction and the event.
21. The system of claim 20 , wherein the association between the emotional reaction and the event is determined based on one or more of the first input, the user data, and the second input.
22. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method comprising:
receiving, at a first time, via a first sensor of a first device, a first input from an environment of a user;
receiving user data via a second device in communication with the first device via a network;
determining, based on the first input and the user data, an occurrence of an event in the environment;
receiving, via a second sensor of the first device, a second input from the user;
identifying, based on the second input, an emotional reaction of the user;
receiving, via the second sensor at a second time later than the first time, a third input from the user;
in accordance with a determination that the third input comprises a request for information about the event:
constructing a query based on the third input, wherein the query is associated with the event and further associated with the requested information; and
presenting a response to the query via a display; and
in accordance with a determination that the third input does not comprise a request for information about the event, forgoing constructing the query.