Alternate avatar controls
Controlling an avatar in an XR environment can be based on input(s) from external, non-XR computing devices, such as mobile devices. An external device can capture sensor data, audio data, and/or image data, which are processed by an avatar control system to determine a gesture performed by the user, gauge the level of intensity or excitement of the user, and/or infer the user's sentiment or emotional state. The avatar control system can generate a motion plan based on these inferences drawn from the external device sensor data based on a baseline animation or motion template. The system can generate avatar animations that have exaggerated or understated expressions relative to the user's real-world gestures, possibly exceeding limitations of a kinematic model that otherwise constrain the avatar's animations. The technology can enable users to control an avatar in an XR environment without specialized XR equipment.
1. A method for controlling an avatar in a virtual environment, the method comprising:
receiving, from a mobile device associated with a user, sensor data captured by one or more sensors of the mobile device, wherein the sensor data is based on movement of the mobile device and wherein the mobile device provides a view into the virtual environment, and wherein the one or more sensors includes a microphone;
selecting a motion template from a plurality of motion templates based on the sensor data, wherein the motion template defines a baseline animation, wherein the motion template is configurable using one or more parameters to modify the baseline animation, and wherein selecting the motion template comprises classifying audio data, from the microphone, according to a model;
computing an intensity score based at least in part on the received sensor data, wherein the intensity score is indicative of a level of excitement of the user;
generating a motion plan based at least in part on the selected motion template and the computed intensity score, wherein the motion plan defines an animation for an avatar; and
causing the avatar to perform the animation according to the generated motion plan.
2. The method of claim 1 , wherein the one or more sensors includes an accelerometer, wherein the sensor data includes acceleration data, and wherein selecting the motion template comprises classifying the acceleration data using a model to determine a gesture.
3. The method of claim 1 , wherein the one or more sensors includes an accelerometer, wherein the sensor data includes acceleration data and wherein computing the intensity score comprises:
determining one or more peak-to-peak amplitude values from the acceleration data in at least one dimension;
normalizing the determined one or more peak-to-peak amplitude values; and
determining the intensity score based at least in part on the normalized one or more peak-to-peak amplitude values.
4. The method of claim 1 , wherein the one or more sensors includes a microphone, wherein the sensor data includes audio data and wherein computing the intensity score comprises:
calculating an average loudness across at least a portion of the audio data;
normalizing the calculated average loudness based on a predetermined mapping; and
determining the intensity score based at least in part on the normalized average loudness.
5. The method of claim 1 , wherein the one or more sensors includes a camera, wherein the sensor data includes image data, and wherein selecting the motion template comprises detecting a user pose based on the image data.
6. The method of claim 1 , wherein the one or more sensors includes a camera, wherein the sensor data includes image data, and wherein computing the intensity score comprises:
calculating a cumulative movement metric across at least a portion of the image data;
normalizing the cumulative movement metric based on a predetermined mapping; and
determining the intensity score based at least in part on the normalized cumulative movement metric.
7. The method of claim 1 , further comprising:
generating, based on the received sensor data and a sensor fusion algorithm, inferred kinematic data,
wherein selecting the motion template is further based on the inferred kinematic data.
8. The method of claim 1 , further comprising:
obtaining user preferences that at least define an intensity score threshold;
determining that the computed intensity score exceeds the intensity score threshold; and
modifying the computed intensity score to be at or below the intensity score threshold defined by the user preferences.
9. The method of claim 1 , further comprising:
determining that the intensity score exceeds a threshold intensity score,
wherein generating the motion plan is further based on the determination that the intensity score exceeds the threshold intensity score, and
wherein the motion plan defines the animation for the avatar to exceed a set of kinematic rules associated with a body model of the avatar, wherein the set of kinematic rules define a range of movement for each body part of a body model of the avatar.
10. A non-transitory computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for controlling an avatar in a virtual environment, the process comprising:
receiving, from a mobile device associated with a user, sensor data captured by one or more sensors of the mobile device, wherein the one or more sensors includes a microphone;
recognizing a facial expression based at least in part on the received sensor data;
selecting a motion template from a plurality of motion templates based on the recognized facial expression, wherein the motion template defines a baseline animation, and wherein the motion template is configurable using one or more parameters to modify the baseline animation, and wherein selecting the motion template comprises classifying audio data, from the microphone, according to a model;
determining a user sentiment based at least in part on the received sensor data;
generating a motion plan based at least in part on the selected motion template and the determined user sentiment, wherein the motion plan defines an animation for an avatar; and
causing the avatar to perform the animation according to the generated motion plan.
11. The non-transitory computer-readable storage medium of claim 10 , wherein recognizing the facial expression comprises:
detecting at least one phoneme from the audio data; and
determining, based on a phoneme to facial expression mapping, a facial expression based on the at least one detected phoneme.
12. The non-transitory computer-readable storage medium of claim 10 , wherein determining the user sentiment comprises:
generating, using a speech-to-text model, text data based on the audio data; and
determining, using a natural language processing model, the user sentiment based on the text data.
13. The non-transitory computer-readable storage medium of claim 10 , wherein the one or more sensors includes a camera, wherein the sensor data includes image data, and wherein recognizing the facial expression comprises:
detecting a plurality of facial keypoints from the image data; and
classifying the plurality of facial keypoints using a model to determine the facial expression.
14. The non-transitory computer-readable storage medium of claim 10 , wherein the one or more sensors includes a camera, wherein the sensor data includes image data, and wherein recognizing the facial expression comprises classifying the image data using an image classification model to determine the facial expression.
15. A computing system for controlling an avatar in a virtual environment, the computing system comprising:
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform operations comprising:
receiving sensor data captured by one or more sensors of a user's mobile device, wherein the sensor data is based on movement of the mobile device and wherein the mobile device provides a view into the virtual environment, and wherein the one or more sensors includes a microphone;
selecting a motion template from a plurality of motion templates based on the sensor data, wherein the motion template defines a baseline animation, and wherein the motion template is configurable using one or more parameters to modify the baseline animation, and wherein selecting the motion template comprises classifying audio data, from the microphone, according to a model;
computing an intensity score based at least in part on the received sensor data, wherein the intensity score is indicative of a level of excitement of the user;
generating a motion plan based at least in part on the selected motion template and the computed intensity score, wherein the motion plan defines an animation for an avatar; and
causing the avatar to perform the animation according to the generated motion plan.
16. The computing system of claim 15 , wherein the operations further comprise:
generating, based on the received sensor data and a sensor fusion algorithm, inferred kinematic data,
wherein selecting the motion template is further based on the inferred kinematic data.
17. The computing system of claim 15 , wherein the operations further comprise:
obtaining user preferences that at least define an intensity score threshold;
determining that the computed intensity score exceeds the intensity score threshold; and
modifying the computed intensity score to be at or below the intensity score threshold defined by the user preferences.
18. The computing system of claim 15 , wherein the operations further comprise:
determining that the intensity score exceeds a threshold intensity score,
wherein generating the motion plan is further based on the determination that the intensity score exceeds the threshold intensity score, and
wherein the motion plan defines the animation for the avatar to exceed a set of kinematic rules associated with a body model of the avatar, wherein the set of kinematic rules define a range of movement for each body part of a body model of the avatar.