Rendering XR avatars based on acoustical features
In one embodiment, a method includes receiving a voice input having acoustic features from a first client system associated with a first user, determining emotions associated with the voice input based on one or more of the acoustic features by machine-learning models, determining facial features for a first extended-reality (XR) avatar representing the first user based on the emotions, and sending instructions for rendering the first XR avatar representing the first user to a second client system associated with a second user, wherein the first XR avatar is rendered with the determined facial features.
1 . A method comprising:
receiving, from a first client system associated with a first user, a voice input having a plurality of acoustic features;
determining, based on one or more of the plurality of acoustic features by one or more machine-learning models, one or more emotions associated with the voice input;
determining, based on the one or more emotions, one or more facial features for a first extended-reality (XR) avatar representing the first user;
determining whether a latency time for rendering the first XR avatar is greater than a threshold time;
delaying a transmission of one or more audio signals corresponding to the voice input to a second client system by a delay time, wherein the delay time is determined based on the latency time; and
sending, to the second client system associated with a second user, instructions for rendering the first XR avatar representing the first user, wherein the first XR avatar is rendered with the determined one or more facial features.
2 . The method of claim 1 , further comprising:
receiving a visual input corresponding to the voice input, wherein the visual input has a plurality of visual signals, and wherein determining the one or more emotions is further based on one or more of the visual signals.
3 . The method of claim 1 , further comprising:
receiving a motion input corresponding to the voice input, wherein the motion input comprises a plurality of motion signals, and wherein determining the one or more emotions is further based on one or more of the motion signals.
4 . The method of claim 1 , wherein the first user is in a conversation with the second user in an XR environment comprising the first XR avatar.
5 . The method of claim 1 , wherein the one or more machine-learning models comprise an acoustic model.
6 . The method of claim 5 , further comprising:
transforming, by the acoustic model, the voice input to a Mel-scale spectrogram, wherein determining the one or more emotions is further based on the Mel-scale spectrogram.
7 . The method of claim 1 , wherein each of the one or more emotions is associated with one or more of a range, an intensity, or a confidence score.
8 . The method of claim 1 , wherein the one or more emotions are associated with one or more confidence scores, respectively, and wherein the method further comprises:
ranking the one or more emotions based on their confidence scores, wherein determining the one or more facial features is based on a top-ranked emotion.
9 . The method of claim 1 , wherein the one or more emotions are associated with one or more confidence scores, respectively, and wherein the method further comprises:
ranking the one or more emotions based on their confidence scores; and
selecting two or more of the ranked emotions based on their rankings;
wherein determining the one or more facial features is based on the selected two or more emotions.
10 . The method of claim 9 , further comprising:
generating a blended emotion based on the selected two or more emotions, wherein determining the one or more facial features is based on the blended emotion.
11 . The method of claim 1 , wherein determining the one or more facial features comprises:
identifying a plurality of facial landmarks of the first XR avatar; and
mapping the one or more emotions to one or more pre-defined expressive shapes for a face of the first XR avatar based on the plurality of facial landmarks.
12 . The method of claim 1 , further comprising:
generating one or more lip movements using a lip-sync model based on the determined one or more emotions, wherein the first XR avatar is rendered further based on the one or more lip movements.
13 . The method of claim 1 , further comprising:
generating a recommendation of a communication content for the first user based on the determined one or more emotions; and
sending, to the first client system, instructions for presenting the recommendation of the communication content.
14 . The method of claim 1 , further comprising:
sending, to an assistant system executing on the first client system, the determined one or more emotions; and
receiving, from the assistant system, execution results of one or more tasks determined based on the one or more emotions.
15 . The method of claim 1 , further comprising:
generating a transcription for the voice input, wherein determining the one or more emotions is further based on an analysis of the transcription.
16 . The method of claim 1 , further comprising:
accessing a prior emotion state from a dialog state associated with the first user, wherein determining the one or more emotions is further based on the prior emotion state.
17 . The method of claim 16 , wherein the prior emotion state is determined based on one or more of a prior user input from the first user or a prior user input from the second user.
18 . One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
receive, from a first client system associated with a first user, a voice input having a plurality of acoustic features;
determine, based on one or more of the plurality of acoustic features by one or more machine-learning models, one or more emotions associated with the voice input;
determine, based on the one or more emotions, one or more facial features for a first extended-reality (XR) avatar representing the first user;
determine whether a latency time for rendering the first XR avatar is greater than a threshold time;
delay a transmission of one or more audio signals corresponding to the voice input to a second client system by a delay time, wherein the delay time is determined based on the latency time; and
send, to the second client system associated with a second user, instructions for rendering the first XR avatar representing the first user, wherein the first XR avatar is rendered with the determined one or more facial features.
19 . A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
receive, from a first client system associated with a first user, a voice input having a plurality of acoustic features;
determine, based on one or more of the plurality of acoustic features by one or more machine-learning models, one or more emotions associated with the voice input;
determine, based on the one or more emotions, one or more facial features for a first extended-reality (XR) avatar representing the first user;
determine whether a latency time for rendering the first XR avatar is greater than a threshold time;
delay a transmission of one or more audio signals corresponding to the voice input to a second client system by a delay time, wherein the delay time is determined based on the latency time; and
send, to the second client system associated with a second user, instructions for rendering the first XR avatar representing the first user, wherein the first XR avatar is rendered with the determined one or more facial features.
20 . The system of claim 19 , wherein the non-transitory memory further comprises instructions executable by the processors to:
receive a visual input corresponding to the voice input, wherein the visual input has a plurality of visual signals, and wherein determining the one or more emotions is further based on one or more of the visual signals.