Cross-device communication with adaptive avatar interaction

1 . A computer-implemented method comprising:

receiving communication inputs from a first device associated with a first user in a virtual experience, the communication inputs comprising at least a video input;

determining facial landmarks and head orientation of the first user based on the communication inputs;

generating an animation of a three-dimensional (3D) avatar of the first user based on the facial landmarks and the head orientation of the first user;

mapping the head orientation of the first user to an orientation of the first device based on one or more of roll, yaw, and pitch of the orientation of the first device;

adjusting a virtual camera position based on the head orientation of the first user, wherein the virtual camera position is a simulated viewpoint within the virtual experience from which a scene that depicts the 3D avatar of the first user is rendered;

receiving communication inputs from a second device associated with a second user in the virtual experience, the second device having additional features not present on the first device, wherein the additional features include hardware capabilities of the second device that are not available on the first device;

enhancing the animation of the 3D avatar based on the communication inputs from the second device; and

providing the enhanced animation of the 3D avatar to the second device for display on the second device.

2 . The computer-implemented method of claim 1 , wherein the second device is a virtual reality (VR) headset, and the additional features comprise VR-specific inputs and outputs.

3 . The computer-implemented method of claim 1 , further comprising:

receiving a request from the second device to switch from a VR mode to an augmented reality (AR) mode; and

adjusting the enhanced animation of the 3D avatar and the virtual camera position based on the AR mode.

4 . The computer-implemented method of claim 1 , wherein the communication inputs from the first device further comprise audio input, and wherein the method further comprises:

generating facial expressions and mouth movements for the 3D avatar based on the audio input.

5 . The computer-implemented method of claim 1 , wherein adjusting the virtual camera position comprises:

determining a position of shoulders of the first user in relation to the head orientation; and

using the position of the shoulders of the first user to calculate a depth for the animation of the 3D avatar.

6 . The computer-implemented method of claim 1 , wherein the second device is an augmented reality (AR) headset, and the additional features comprise AR-specific functionality comprising at least rendering the 3D avatar in a real-world environment as viewed through the AR headset.

7 . The computer-implemented method of claim 1 , wherein enhancing the animation of the 3D avatar comprises adjusting an appearance and one or more movements of the 3D avatar based on device-specific features of the second device.

8 . The computer-implemented method of claim 1 , wherein the enhanced animation of the 3D avatar comprises a portion of a call in the virtual experience between the first user and the second user.

9 . A system comprising:

one or more processors; and

memory coupled to the one or more processors storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

receiving communication inputs from a first device associated with a first user in a virtual experience, the communication inputs comprising at least a video input;

determining facial landmarks and head orientation of the first user based on the communication inputs;

generating an animation of a three-dimensional (3D) avatar of the first user based on the facial landmarks and the head orientation of the first user;

mapping the head orientation of the first user to an orientation of the first device based on one or more of roll, yaw, and pitch of the orientation of the first device;

adjusting a virtual camera position based on the head orientation of the first user, wherein the virtual camera position is a simulated viewpoint within the virtual experience from which a scene that depicts the 3D avatar of the first user is rendered;

receiving communication inputs from a second device associated with a second user in the virtual experience, the second device having additional features not present on the first device, wherein the additional features include hardware capabilities of the second device that are not available on the first device;

enhancing the animation of the 3D avatar based on the communication inputs from the second device; and

providing the enhanced animation of the 3D avatar to the second device for display on the second device.

10 . The system of claim 9 , wherein the second device is a virtual reality (VR) headset, and the additional features comprise VR-specific inputs and outputs.

11 . The system of claim 9 , wherein the instructions cause the one or more processors to perform further operations comprising:

receiving a request from the second device to switch from a VR mode to an augmented reality (AR) mode; and

adjusting the enhanced animation of the 3D avatar and the virtual camera position based on the AR mode.

12 . The system of claim 9 , wherein the communication inputs from the first device further comprise audio input, and wherein the instructions cause the one or more processors to perform a further operation comprising:

generating facial expressions and mouth movements for the 3D avatar based on the audio input.

13 . The system of claim 9 , wherein adjusting the virtual camera position comprises:

determining a position of shoulders of the first user in relation to the head orientation; and

using the position of the shoulders of the first user to calculate a depth for the animation of the 3D avatar.

14 . The system of claim 9 , wherein the second device is an augmented reality (AR) headset, and the additional features comprise AR-specific functionality comprising at least rendering the 3D avatar in a real-world environment as viewed through the AR headset.

15 . The system of claim 9 , wherein enhancing the animation of the 3D avatar comprises adjusting an appearance and one or more movements of the 3D avatar based on device-specific features of the second device.

16 . The system of claim 9 , wherein the enhanced animation of the 3D avatar comprises a portion of a call in the virtual experience between the first user and the second user.

17 . A non-transitory computer-readable medium with instructions stored thereon that, when executed by a processor, cause the processor to perform operations comprising:

receiving communication inputs from a first device associated with a first user in a virtual experience, the communication inputs comprising at least a video input;

determining facial landmarks and head orientation of the first user based on the communication inputs;

generating an animation of a three-dimensional (3D) avatar of the first user based on the facial landmarks and the head orientation of the first user;

mapping the head orientation of the first user to an orientation of the first device based on one or more of roll, yaw, and pitch of the orientation of the first device;

adjusting a virtual camera position based on the head orientation of the first user, wherein the virtual camera position is a simulated viewpoint within the virtual experience from which a scene that depicts the 3D avatar of the first user is rendered;

receiving communication inputs from a second device associated with a second user in the virtual experience, the second device having additional features not present on the first device, wherein the additional features include hardware capabilities of the second device that are not available on the first device;

enhancing the animation of the 3D avatar based on the communication inputs from the second device; and

providing the enhanced animation of the 3D avatar to the second device for display on the second device.

18 . The non-transitory computer-readable medium of claim 17 , wherein the second device is a virtual reality (VR) headset, and the additional features comprise VR-specific inputs and outputs.

19 . The non-transitory computer-readable medium of claim 17 , wherein the instructions cause the processor to perform further operations comprising:

receiving a request from the second device to switch from a VR mode to an augmented reality (AR) mode; and

adjusting the enhanced animation of the 3D avatar and the virtual camera position based on the AR mode.

20 . The non-transitory computer-readable medium of claim 17 , wherein the communication inputs from the first device further comprise audio input, and wherein the instructions cause the processor to perform a further operation comprising:

generating facial expressions and mouth movements for the 3D avatar based on the audio input.