Extended reality assistance based on user understanding
Various implementations disclosed herein provide augmentations in extended reality (XR) using sensor data from a user worn device. The sensor data may be used understand that a user's state is associated with providing user assistance, e.g., a user's appearance or behavior or an understanding of the environment may be used to recognize a need or desire for user assistance. The augmentations may assist the user by enhancing or supplementing the user's abilities, e.g., providing guidance or other information about an environment to disabled/impaired person.
1 . A method comprising:
at an electronic device having a processor;
capturing sensor data while the electronic device is worn by a user in a physical environment, wherein a user assistance feature is enabled on the device to assist with physical environment understanding based on a user state;
in accordance with the assistance feature being enabled:
determining the user state in the physical environment based on monitoring the sensor data, wherein the user state corresponds to a user requirement for assistance in the physical environment; and
presenting content based on the user state, the content at least partially based on the user requirement for assistance in the physical environment, wherein the content provides the assistance by guiding the user to move within the physical environment to a specific location.
2 . The method of claim 1 , wherein the user state comprises a user familiarity with the physical environment, and wherein the user familiarity with the physical environment is determined based on:
assessing data indicative of whether the user has previously visited the physical environment; or
identifying uncertainty, hesitation, or confusion based on a posture, a behavior, or an expression of the user.
3 . The method of claim 1 , wherein determining the user state is based on determining an orientation of a torso of the user in the physical environment.
4 . The method of claim 1 , wherein determining the user state is based on identifying a current task of the user.
5 . The method of claim 1 , wherein the content guides the user to move from a current location to a destination within the physical environment.
6 . The method of claim 1 , wherein the content guides the user based on:
identifying an orientation of a torso of the user relative to the physical environment; and
instructions provided relative to the orientation of the torso of the user relative to the physical environment.
7 . The method of claim 1 , wherein the sensor data comprises images of an eye region and a mouth region of the user, and wherein the method further comprises recognizing an expression of the user based on the sensor data, wherein the user state is determined based on recognizing the expression.
8 . The method of claim 1 , wherein the sensor data comprises images of a hand of the user, and wherein the method further comprises recognizing a gesture of the user based on the sensor data, wherein the user state is determined based on recognizing the gesture.
9 . A device comprising:
a non-transitory computer-readable storage medium; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the one or more processors to perform operations comprising:
capturing sensor data while the device is worn by a user in a physical environment, wherein a user assistance feature is enabled on the device to assist with physical environment understanding based on a user state;
in accordance with the assistance feature being enabled:
determining the user state in the physical environment based on monitoring the sensor data, wherein the user state corresponds to a user requirement for assistance in the physical environment; and
presenting content based on the user state, the content at least partially based on the user requirement for assistance in the physical environment, wherein the content provides the assistance by guiding the user to move within the physical environment to a specific location.
10 . The device of claim 9 , wherein the user state comprises a user familiarity with the physical environment, and wherein the user familiarity with the physical environment is determined based on:
assessing data indicative of whether the user has previously visited the physical environment; or
identifying uncertainty, hesitation, or confusion based on a posture, a behavior, or an expression of the user.
11 . The device of claim 9 , wherein determining the user state is based on determining an orientation of a torso of the user in the physical environment.
12 . The device of claim 9 , wherein determining the user state is based on identifying a current task of the user.
13 . The device of claim 9 , wherein the content guides the user to move from a current location to a destination within the physical environment.
14 . The device of claim 9 , wherein the content guides the user based on:
identifying an orientation of a torso of the user relative to the physical environment; and
instructions provided relative to the orientation of the torso of the user relative to the physical environment.
15 . The device of claim 9 , wherein the sensor data comprises images of an eye region and a mouth region of the user, and wherein the operations further comprises recognizing an expression of the user based on the sensor data, wherein the user state is determined based on recognizing the expression.
16 . The device of claim 9 , wherein the sensor data comprises images of a hand of the user, and wherein the method further comprises recognizing a gesture of the user based on the sensor data, wherein the user state is determined based on recognizing the gesture.
17 . A non-transitory computer-readable storage medium, storing program instructions executable by one or more processors to perform operations comprising:
capturing sensor data while an electronic device is worn by a user in a physical environment, wherein a user assistance feature is enabled on the electronic device to assist with physical environment understanding based on a user state;
in accordance with the assistance feature being enabled:
determining the user state in the physical environment based on monitoring the sensor data, wherein the user state corresponds to a user requirement for assistance in the physical environment; and
presenting content based on the user state, the content at least partially based on the user requirement for assistance in the physical environment, wherein the content provides the assistance by guiding the user to move within the physical environment to a specific location.
18 . The non-transitory computer-readable storage medium of claim 17 , wherein the user state comprises a user familiarity with the physical environment, and wherein the user familiarity with the physical environment is determined based on:
assessing data indicative of whether the user has previously visited the physical environment; or
identifying uncertainty, hesitation, or confusion based on a posture, a behavior, or an expression of the user.
19 . The non-transitory computer-readable storage medium of claim 17 , wherein determining the user state is based on determining an orientation of a torso of the user in the physical environment.
20 . The non-transitory computer-readable storage medium of claim 17 , wherein determining the user state is based on identifying a current task of the user.
21 . The non-transitory computer-readable storage medium of claim 17 , wherein the content guides the user to move from a current location to a destination within the physical environment.
22 . The non-transitory computer-readable storage medium of claim 17 , wherein the content guides the user based on:
identifying an orientation of a torso of the user relative to the physical environment; and
instructions provided relative to the orientation of the torso of the user relative to the physical environment.
23 . The non-transitory computer-readable storage medium of claim 17 , wherein the sensor data comprises images of an eye region and a mouth region of the user, and wherein the method further comprises recognizing an expression of the user based on the sensor data, wherein the user state is determined based on recognizing the expression.
24 . The non-transitory computer-readable storage medium of claim 17 , wherein the sensor data comprises images of a hand of the user, and wherein the method further comprises recognizing a gesture of the user based on the sensor data, wherein the user state is determined based on recognizing the gesture.