Camera reprojection for faces

1. A method comprising, by one or more computing systems:

receiving an image of a portion of a face of a first user, wherein the image is captured by a camera of a first artificial-reality system worn by the first user;

accessing a three-dimensional (3D) facial model representative of the face of the first user;

identifying one or more facial features captured in the image;

determining a relative pose of the camera of the first artificial-reality system relative to the 3D facial model based on comparison of the identified one or more facial features in the image and predetermined feature locations on the 3D facial model;

determining a mapping relationship between the image captured by the camera of the first artificial-reality system and the 3D facial model by projecting the image of the portion of the face of the first user onto the 3D facial model from the relative pose of the camera that captured the image;

sending a rendering package to a second artificial-reality system worn by a second user, the rendering package including (1) the 3D facial model of the first user, (2) the mapping relationship between the image captured by the camera of the first artificial-reality system worn by the first user and the 3D facial model of the first user, and (3) instructions to render a facial representation of the first user based on the 3D facial model and the mapping relationship; and

causing an output image of the facial representation of the first user to be rendered from a viewpoint of the second user with respect to the first user using the rendering package.

2. The method of claim 1 , further comprising:

receiving a plurality of images corresponding to a second portion of the face of the first user, wherein the plurality of images is captured by a plurality of cameras coupled to the first artificial-reality system worn by the first user;

synthesizing, using a machine-learning model, to generate a synthesized image corresponding to the second portion of the face of the first user; and

determining a second mapping relationship between the synthesized image and the 3D facial model by projecting the synthesized image of the second portion of the face of the first user onto the 3D facial model from a predetermined camera pose.

3. The method of claim 1 , wherein the 3D facial model is a predetermined 3D facial model representative of a plurality of faces of a plurality of users.

4. The method of claim 1 , wherein the 3D facial model is generated by:

morphing, based on at least the identified one or more facial features in the image, a predetermined 3D facial model representative of a plurality of faces of a plurality of users.

5. The method of claim 1 , wherein determining the relative pose of the camera of the first artificial-reality system comprises comparing locations of the identified one or more facial features captured in the image to the predetermined feature locations.

6. The method of claim 1 , wherein the output image of the facial representation of the first user is photorealistic.

7. The method of claim 1 , further comprising:

receiving a second image of a second portion of the face of the first user, wherein the second image is captured by a second camera of the first artificial-reality system worn by the first user;

identifying one or more second facial features captured in the second image;

determining a second relative pose of the second camera relative to the 3D facial model based on the identified one or more second facial features in the second image and the predetermined feature locations on the 3D facial model; and

determining a second mapping relationship between the second image and the 3D facial model by projecting the second image of the second portion of the face of the first user onto the 3D facial model from the second relative pose of the second camera that captured the second image, wherein the output image further uses at least the 3D facial model and the second mapping relationship between the second image and the 3D facial model.

8. The method of claim 1 , wherein the mapping relationship is a texture image of the portion of the face of the first user, and wherein the texture image is blended with a predetermined texture corresponding to other portions of the face of the first user to generate the output image for the facial representation of the first user.

9. The method of claim 1 , wherein the mapping relationship is a texture image of the portion of the face of the first user, and wherein rendering the facial representation of the first user comprises:

sampling a first point on a predetermined texture corresponding to the facial representation of the first user to identify a first color associated with the first point;

sampling a second point on the texture image that corresponds to the first point on the predetermined texture to identify a second color associated with the second point; and

blending the first color of the first point with the second color of the second point to generate a final color associated with a location corresponding to the first point and the second point.

10. The method of claim 1 , wherein the first artificial-reality system worn by the first user or the second artificial-reality system worn by the second user is an artificial-reality headset.

11. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

receive an image of a portion of a face of a first user, wherein the image is captured by a camera of a first artificial-reality system worn by the first user;

access a three-dimensional (3D) facial model representative of the face of the first user;

identify one or more facial features captured in the image;

determine a relative pose of the camera of the first artificial-reality system relative to the 3D facial model based on comparison of the identified one or more facial features in the image and predetermined feature locations on the 3D facial model;

determine a mapping relationship between the image captured by the camera of the first artificial-reality system and the 3D facial model by projecting the image of the portion of the face of the first user onto the 3D facial model from the relative pose of the camera that captured the image;

send a rendering package to a second artificial-reality system worn by a second user, the rendering package including (1) the 3D facial model of the first user, (2) the mapping relationship between the image captured by the camera of the first artificial-reality system worn by the first user and the 3D facial model of the first user, and (3) instructions to render a facial representation of the first user based on the 3D facial model and the mapping relationship; and

cause an output image of the facial representation of the first user to be rendered from a viewpoint of the second user with respect to the first user using the rendering package.

12. The media of claim 11 , wherein the software is further operable when executed to:

receive a plurality of images corresponding to a second portion of the face of the first user, wherein the plurality of images is captured by a plurality of cameras coupled to the first artificial-reality system worn by the first user;

synthesize, using a machine-learning model, to generate a synthesized image corresponding to the second portion of the face of the first user; and

determine a second mapping relationship between the synthesized image and the 3D facial model by projecting the synthesized image of the second portion of the face of the first user onto the 3D facial model from a predetermined camera pose.

13. The media of claim 11 , wherein the 3D facial model is a predetermined 3D facial model representative of a plurality of faces of a plurality of users.

14. The media of claim 11 , wherein the software is further operable when executed to:

morph, based on at least the identified one or more facial features in the image, a predetermined 3D facial model representative of a plurality of faces of a plurality of users.

15. The media of claim 11 , wherein determining the relative pose of the camera of the first artificial-reality system comprises comparing locations of the identified one or more facial features captured in the image to the predetermined feature locations.

16. A system comprising:

one or more processors; and

one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to:

receive an image of a portion of a face of a first user, wherein the image is captured by a camera of a first artificial-reality system worn by the first user;

access a three-dimensional (3D) facial model representative of the face of the first user;

identify one or more facial features captured in the image;

determine a relative pose of the camera of the first artificial-reality system relative to the 3D facial model based on comparison of the identified one or more facial features in the image and predetermined feature locations on the 3D facial model;

determine a mapping relationship between the image captured by the camera of the first artificial-reality system and the 3D facial model by projecting the image of the portion of the face of the first user onto the 3D facial model from the relative pose of the camera that captured the image;

send a rendering package to a second artificial-reality system worn by a second user, the rendering package including (1) the 3D facial model of the first user, (2) the mapping relationship between the image captured by the camera of the first artificial-reality system worn by the first user and the 3D facial model of the first user, and (3) instructions to render a facial representation of the first user based on the 3D facial model and the mapping relationship; and

cause an output image of the facial representation of the first user to be rendered from a viewpoint of the second user with respect to the first user using the rendering package.

17. The system of claim 16 , wherein the processors are further operable when executing the instructions to:

receive a plurality of images corresponding to a second portion of the face of the first user, wherein the plurality of images is captured by a plurality of cameras coupled to the first artificial-reality system worn by the first user;

synthesize, using a machine-learning model, to generate a synthesized image corresponding to the second portion of the face of the first user; and

determine a second mapping relationship between the synthesized image and the 3D facial model by projecting the synthesized image of the second portion of the face of the first user onto the 3D facial model from a predetermined camera pose.

18. The system of claim 16 , wherein the 3D facial model is a predetermined 3D facial model representative of a plurality of faces of a plurality of users.

19. The system of claim 16 , wherein the processors are further operable when executing the instructions to:

morph, based on at least the identified one or more facial features in the image, a predetermined 3D facial model representative of a plurality of faces of a plurality of users.

20. The system of claim 16 , wherein determining the relative pose of the camera of the first artificial-reality system comprises comparing locations of the identified one or more facial features captured in the image to the predetermined feature locations.