Photoreal character configurations for spatial computing
Systems and methods for displaying a virtual character in a mixed reality environment are disclosed. In some embodiments, a view of the virtual character is based on an animation rig comprising primary joints and helper joints. The animation rig may be in a pose defined by spatial relationships between the primary joints and helper joints. The virtual character may be moving in the mixed reality environment. In some instances, the virtual character may be moving based on a comparison of interestingness values associated with elements in the mixed reality environment. The spatial relationship transformation associated with the movement may be indicated by movement information. In some embodiments, the movement information is received from a neural network.
1. A method comprising:
displaying, on a display of a wearable head device, a view of a character in a mixed reality environment, wherein:
the view of the character is based on an animation rig associated with the character;
the animation rig comprises primary joints and helper joints,
the primary joints are associated with skeletal joints of the character,
the helper joints are associated with locations of the animation rig, and
at a first time, the animation rig is in a first pose defined by first spatial relationships between the primary joints and the helper joints;
determining, at a second time, that the animation rig is moving from the first pose to a second pose, wherein the second pose is defined by second spatial relationships between the primary joints and the helper joints, wherein determining that the animation rig is moving from the first pose to the second pose comprises determining that an interestingness value associated with a first element of the mixed reality environment exceeds an interestingness value associated with a second element of the mixed reality environment; and
updating a display, on the display of the wearable head device, of the animation rig from the first pose to the second pose based on movement information, wherein the movement information indicates a transformation from the first spatial relationships to the second spatial relationships.
2. The method of claim 1 , further comprising:
transmitting, to a neural network, information associated with a movement of the animation rig; and
receiving, from the neural network, the movement information, wherein the movement information is based on the transmitted information.
3. The method of claim 2 , wherein the neural network uses an ReLU activation function to compute the movement information.
4. The method of claim 2 , wherein the movement information is computed based on training data presented to the neural network.
5. The method of claim 1 , wherein the locations of the helper joints are determined by a helper joint placement criterion comprising one or more of a manual definition and a maximum movement criterion.
6. The method of claim 1 , wherein the movement information is determined using a regression analysis between the primary joints and the helper joints to compute the second spatial relationships.
7. The method of claim 1 , wherein determining that the animation rig is moving from the first pose to the second pose comprises determining the animation rig is in the first pose longer than a character discomfort threshold time.
8. The method of claim 1 , wherein determining that the animation rig is moving from the first pose to the second pose comprises determining a line of sight associated with the wearable head device has changed.
9. The method of claim 1 , wherein the character is displayed to be positioned on a surface of the mixed reality environment, the method further comprising:
sensing, with a sensor of the wearable head device, the surface of the mixed reality environment; and
aligning the animation rig and the surface based on the sensed surface, wherein the first pose is determined based on the alignment of the animation rig and the surface.
10. The method of claim 1 , wherein the transformation from the first spatial relationships to the second spatial relationships is associated with the character's muscle movements when the animation rig moves from the first pose to the second pose.
11. The method of claim 1 , wherein:
the movement information is represented at least in part in quaternions, and
a loss function associated with the movement information comprises combining mean squared error loss of quaternion component values and a penalty for the component values.
12. The method of claim 1 , wherein displaying the view of the character in the mixed reality environment comprises rendering the view of the character using Forward+.
13. The method of claim 1 , wherein the first and second elements are each associated with one of a gaze box associated with an inanimate object of the mixed reality environment, an interesting area associated with an event of the mixed reality environment, and a social triangle associated with a user in the mixed reality environment.
14. A system comprising:
a wearable head device comprising a display; and
one or more processors configured to execute a method comprising:
displaying, on the display, a view of a character in a mixed reality environment, wherein:
the view of the character is based on an animation rig associated with the character;
the animation rig comprises primary joints and helper joints,
the primary joints are associated with skeletal joints of the character,
the helper joints are associated with locations of the animation rig, and
at a first time, the animation rig is in a first pose defined by first spatial relationships between the primary joints and the helper joints;
determining, at a second time, that the animation rig is moving from the first pose to a second pose, wherein the second pose is defined by second spatial relationships between the primary joints and the helper joints, wherein determining that the animation rig is moving from the first pose to the second pose comprises determining that an interestingness value associated with a first element of the mixed reality environment exceeds an interestingness value associated with a second element of the mixed reality environment; and
updating a display, on the display, of the animation rig from the first pose to the second pose based on movement information, wherein the movement information indicates a transformation from the first spatial relationships to the second spatial relationships.
15. The system of claim 14 , wherein the method further comprises:
transmitting, to a neural network, information associated with a movement of the animation rig; and
receiving, from the neural network, the movement information, wherein the movement information is based on the transmitted information.
16. The system of claim 14 , wherein the locations of the helper joints are determined by a helper joint placement criterion comprising one or more of a manual definition and a maximum movement criterion.
17. The system of claim 14 , wherein determining that the animation rig is moving from the first pose to the second pose comprises determining a line of sight associated with the wearable head device has changed.
18. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to execute a method comprising:
displaying, on a display of a wearable head device, a view of a character in a mixed reality environment, wherein:
the view of the character is based on an animation rig associated with the character;
the animation rig comprises primary joints and helper joints,
the primary joints are associated with skeletal joints of the character,
the helper joints are associated with locations of the animation rig, and
at a first time, the animation rig is in a first pose defined by first spatial relationships between the primary joints and the helper joints;
determining, at a second time, that the animation rig is moving from the first pose to a second pose, wherein the second pose is defined by second spatial relationships between the primary joints and the helper joints, wherein determining that the animation rig is moving from the first pose to the second pose comprises determining that an interestingness value associated with a first element of the mixed reality environment exceeds an interestingness value associated with a second element of the mixed reality environment; and
updating a display, on the display of the wearable head device, of the animation rig from the first pose to the second pose based on movement information, wherein the movement information indicates a transformation from the first spatial relationships to the second spatial relationships.
19. The non-transitory computer-readable medium of claim 18 , wherein the method further comprises:
transmitting, to a neural network, information associated with a movement of the animation rig; and
receiving, from the neural network, the movement information, wherein the movement information is based on the transmitted information.
20. The non-transitory computer-readable medium of claim 18 , wherein the locations of the helper joints are determined by a helper joint placement criterion comprising one or more of a manual definition and a maximum movement criterion.