Contextual-based rendering of virtual avatars
Examples of systems and methods for rendering an avatar in a mixed reality environment are disclosed. The systems and methods may be configured to automatically scale an avatar or to render an avatar based on a determined intention of a user, an interesting impulse, environmental stimuli, or user saccade points. The disclosed systems and methods may apply discomfort curves when rendering an avatar. The disclosed systems and methods may provide a more realistic interaction between a human user and an avatar.
1. A wearable device for determining an intent of an user interaction, the wearable device comprising:
a device sensor configured to acquire interaction data regarding interactions of a user with an environment of the user; and
a hardware processor programmed to:
determine an object of interest in the environment of the user, wherein to determine the object of interest the hardware processor:
extends a vector from the user towards each potential object of interest and a sight line vector that extends from the user outwards directly in front of the user;
assigns an interest weight value to each potential object of interest, wherein the interest weight of a potential object of interest dynamically changes to reflect changes in the environment; and
calculates an overall interest value for each potential object of interest, wherein the overall interest value for a potential object of interest is based at least partly on the interest weight value for the potential object of interest;
for each potential object of interest:
computing a dot product between the vector that extends from the user towards the potential object of interest and the sight line vector; and
multiplying the dot product by the interest weight value assigned to the potential object of interest to calculate the overall interest value for the potential object of interest; and
determining that the potential object of interest having a largest overall interest value is the object of interest;
identify a user interaction based on the interaction data acquired from the device sensor, wherein the user interaction includes at least one of an interaction with the object of interest in the environment of the user;
decompose the user interaction into at least one of: a world component or a local component, wherein the world component includes an action with reference to a world frame of the environment and the local component includes an action with reference to a local frame of a body of the user;
determine an intent associated with the user interaction based on contextual information, wherein the contextual information includes a past activity of the user; and
communicate at least one of: the intent, information associated with the world component, or information associated with the local component to another wearable device to cause the other wearable device to map the user interaction to an avatar of the user rendered by the other wearable device based on the intent.
2. The wearable device of claim 1 , wherein the object of interest includes a physical object in the environment or a virtual object.
3. The wearable device of claim 1 , wherein the object of interest is a virtual avatar of another user.
4. The wearable device of claim 1 , wherein the user interaction is performed via at least one of a head pose, eye gaze, or body pose, or a movement in the environment of the user.
5. The wearable device of claim 1 , wherein the device sensor includes at least one of: an inertial measurement unit (IMU), an outward-facing camera, or an eye tracking camera.
6. The wearable device of claim 1 , wherein the contextual information includes at least one of a layout of objects in the environment of the user, a location of the objects in the environment, a position of the user, or an orientation of the user.
7. The wearable device of claim 1 , wherein the intent is associated with the world component of the user interaction.
8. The wearable device of claim 1 , wherein the information associated with the local component includes an animation of the local component of the user interaction.
9. A method for rendering a virtual avatar for a remote user interaction, the method comprising:
determining contextual information associated with a first environment of a first user in which an avatar of a second user is rendered, wherein the contextual information includes a past activity of the first user;
determining an object of interest in a second environment of the second user, wherein the determining the object of interest in the second environment of the second user includes:
extending a vector from the second user towards each potential object of interest and a sight line vector that extends from the second user outwards directly in front of the second user;
assigning an interest weight value to each potential object of interest, wherein the interest weight of a potential object of interest dynamically changes to reflect changes in the second environment of the second user; and
calculating an overall interest value for each potential object of interest, wherein the overall interest value for a potential object of interest is based at least partly on the interest weight value for the potential object of interest; and
for each potential object of interest:
computing a dot product between the vector that extends from the second user towards the potential object of interest and the sight line vector; and
multiplying the dot product by the interest weight value assigned to the potential object of interest to calculate the overall interest value for the potential object of interest; and
determining that the potential object of interest having a largest overall interest value is the object of interest;
determining an intent of the second user associated with a user interaction of the second user that occurred in the second environment of the second user, wherein the intent of the second user includes an intent to interact with the object of interest;
determining a world action of the avatar of the second user with respect to a world frame of the first user based on contextual information and the intent of the second user;
animating a local action of the avatar of the second user with respect to a local frame of the avatar of the second user based on a local component of the user interaction of the second user that occurred in the second environment; and
causing a display to render the avatar of the second user including the world action and the local action in a mixed reality environment.
10. The method of claim 9 , wherein determining the intent of the second user includes at least one of: receiving intent data from a wearable device of the second user; or determining the intent of the second user from interaction data of the user interaction of the second user.
11. The method of claim 9 , wherein the intent of the second user includes an intent to communicate or interact with the first user.
12. The method of claim 9 , wherein the local action is further animated based on the intent of the second user associated with the local action.
13. The method of claim 9 , wherein animating the local action includes performing a collision detection of the avatar as a result of the local action and objects in the first environment.
14. The method of claim 9 , wherein the world action includes at least a movement of the avatar of the second user in the first environment or an interaction by the avatar of the second user with an object in the first environment.