Brain-aware extended reality
An extended Reality (XR) system is provided that monitors neurological signals to determine an engagement of a user with a real-world environment. The XR system continuously monitors neurological signals of a user through a processor operating in a low-power mode. The XR system generates an engagement signal by analyzing endogenous brain patterns in the neurological signals. In response to the engagement signal, the XR system activates environmental sensors to capture real-world environment data. The XR system generates contextual data from the captured environment data and determines XR content to provide to the user based on the contextual data. The XR system selectively activates XR capabilities to display the determined XR content.
1 . A machine-implemented method comprising:
continuously monitoring neurological signals of a user by at least one processor of an extended Reality (XR) system operating in a low-power mode;
generating an engagement signal using the neurological signals;
responsive to generating the engagement signal, performing operations comprising:
capturing real-world environment data by activating at least one environmental sensor to capture the real-world environment data of a real-world environment;
generating contextual data using the real-world environment data;
determining XR content to be displayed to the user using the contextual data; and
selectively activating one or more XR capabilities of the XR system to display XR content.
2 . The method of claim 1 , wherein generating the engagement signal comprises:
tracking eye movements of the user;
detecting an intentional eye movement pattern using the eye movements; and
generating the engagement signal using the intentional eye movement pattern and the neurological signals.
3 . The method of claim 1 , wherein generating the engagement signal comprises:
using a first Machine Learning (ML) model to generate a continuous engagement value;
using a second ML model to detect a user intent as a brain-based click; and
generating the engagement signal using the continuous engagement value and the brain-based click.
4 . The method of claim 1 , wherein determining the XR content comprises:
determining an identification of a physical object in the real-world environment using the real-world environment data; and
generating the contextual data using the identification of the physical object.
5 . The machine-implemented method of claim 1 , further comprising:
providing feedback to the user prior to activating the XR capabilities; and
analyzing a neurological response to the feedback to confirm user intent using the neurological signals.
6 . The machine-implemented method of claim 1 , further comprising:
detecting a presence of an additional user using an additional XR system; and
analyzing respective engagement signals and real-world environment data of the XR system and the additional XR system to generate a shared engagement signal.
7 . The machine-implemented method of claim 1 , wherein the XR system is a head-wearable apparatus.
8 . A machine comprising:
at least one processor; and
at least one memory storing instructions that, when executed by the at least one processor, cause the machine to perform operations comprising:
continuously monitoring neurological signals of a user by the at least one processor operating in a low-power mode;
generating an engagement signal using the neurological signals;
responsive to generating the engagement signal, performing operations comprising:
capturing real-world environment data by activating at least one environmental sensor to capture the real-world environment data of a real-world environment;
generating contextual data using the real-world environment data;
determining extended Reality (XR) content to be displayed to the user using the contextual data; and
selectively activating one or more XR capabilities of an XR system to display XR content.
9 . The machine of claim 8 , wherein generating the engagement signal comprises:
tracking eye movements of the user;
detecting an intentional eye movement pattern using the eye movements; and
generating the engagement signal using the intentional eye movement pattern and the neurological signals.
10 . The machine of claim 8 , wherein generating the engagement signal comprises:
using a first Machine Learning (ML) model to generate a continuous engagement value;
using a second ML model to detect a user intent as a brain-based click; and
generating the engagement signal using the continuous engagement value and the brain-based click.
11 . The machine of claim 8 , wherein determining the XR content comprises:
determining an identification of a physical object in the real-world environment using the real-world environment data; and
generating the contextual data using the identification of the physical object.
12 . The machine of claim 8 , wherein the operations further comprise:
providing feedback to the user prior to activating the XR capabilities; and
analyzing a neurological response to the feedback to confirm user intent using the neurological signals.
13 . The machine of claim 8 , wherein the operations further comprise:
detecting a presence of an additional user using an additional XR system; and
analyzing respective engagement signals and real-world environment data of the XR system and the additional XR system to generate a shared engagement signal.
14 . The machine of claim 8 , wherein the XR system is a head-wearable apparatus.
15 . A machine-storage medium, the machine-storage medium including instructions that, when executed by a machine, cause the machine to perform operations comprising:
continuously monitoring neurological signals of a user by at least one processor of an extended Reality (XR) system operating in a low-power mode;
generating an engagement signal using the neurological signals;
responsive to generating the engagement signal, performing operations comprising:
capturing real-world environment data by activating at least one environmental sensor to capture the real-world environment data of a real-world environment;
generating contextual data using the real-world environment data;
determining XR content to be displayed to the user using the contextual data; and
selectively activating one or more XR capabilities of the XR system to display XR content.
16 . The machine-storage medium of claim 15 , wherein generating the engagement signal comprises:
tracking eye movements of the user;
detecting an intentional eye movement pattern using the eye movements; and
generating the engagement signal using the intentional eye movement pattern and the neurological signals.
17 . The machine-storage medium of claim 15 , wherein generating the engagement signal comprises:
using a first Machine Learning (ML) model to generate a continuous engagement value;
using a second ML model to detect a user intent as a brain-based click; and
generating the engagement signal using the continuous engagement value and the brain-based click.
18 . The machine-storage medium of claim 15 , wherein determining the XR content comprises:
determining an identification of a physical object in the real-world environment using the real-world environment data; and
generating the contextual data using the identification of the physical object.
19 . The machine-storage medium of claim 15 , wherein the operations further comprise:
providing feedback to the user prior to activating the XR capabilities; and
analyzing a neurological response to the feedback to confirm user intent using the neurological signals.
20 . The machine-storage medium of claim 15 , wherein the XR system is a head-wearable apparatus.