Holographic calling for artificial reality
A holographic calling system can capture and encode holographic data at a sender-side of a holographic calling pipeline and decode and present the holographic data as a 3D representation of a sender at a receiver-side of the holographic calling pipeline. The holographic calling pipeline can include stages to capture audio, color images, and depth images; densify the depth images to have a depth value for each pixel while generating parts masks and a body model; use the masks to segment the images into parts needed for hologram generation; convert depth images into a 3D mesh; paint the 3D mesh with color data; perform torso disocclusion; perform face reconstruction; and perform audio synchronization. In various implementations, different of these stages can be performed sender-side or receiver side. The holographic calling pipeline also includes sender-side compression, transmission over a communication channel, and receiver-side decompression and hologram output.
1. A method for performing a facial reconstruction on a representation of a sending user of a holograph call, the method comprising:
producing a version of the representation of the sending user, without an XR headset, by applying a facial reconstruction model in the holographic call to the representation of the sending user, wherein the representation of the sending user depicts an XR headset;
wherein the facial reconstruction model is trained by:
generating a set of training items, each training item generated by pairing an image of a user that does not depict an XR headset with a version of that image of the user with the XR headset superimposed; and
applying each particular training item, of the set of training items, to the facial reconstruction model by:
encoding the version of the image of the user with the XR headset superimposed, from the particular training item;
generating a predicted geometry by applying a geometry prediction branch of the facial reconstruction model to the encoded version of the image;
updating parameters of the geometry prediction branch based on a comparison of the predicted geometry and a geometry generated for the image of the user without the XR headset;
generating a predicted texture by applying a texture prediction branch of the facial reconstruction model to the encoded version of the image; and
updating parameters of the texture prediction branch based on a comparison of the predicted texture and a texture generated for the image of the user without the XR headset; and
wherein applying the facial reconstruction model includes:
encoding the representation of the sending user;
generating a predicted geometry by applying the geometry prediction branch of the facial reconstruction model to the encoded version of the representation of the sending user;
generating a predicted texture by applying the texture prediction branch of the facial reconstruction model to the encoded version of the representation of the sending user; and
skinning the predicted texture onto the predicted geometry.
2. The method of claim 1 , wherein the encoding is performed by a convolutional neural network.
3. The method of claim 1 , wherein the texture branch includes a decoder, that is a neural network with deconvolution layers having skip connections, that modifies an encoded input image to shape it for fitting onto a facial geometry.
4. The method of claim 1 wherein the geometry prediction branch includes a neural network that receives an encoded image and produces a predicted geometry of the face depicted in the encoded image.
5. The method of claim 1 ,
wherein updating parameters of the geometry prediction branch includes applying multiple loss functions for L2 regularization, weighted parameter distance cost, and 3D vertex cost; and
wherein updating parameters of the texture prediction branch includes applying multiple loss functions for L1 loss, SSIM loss, and/or perceptual loss.
6. A non-transitory computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for performing a facial reconstruction on a representation of a sending user of a holograph call, the process comprising:
producing a version of the representation of the sending user without an XR headset by applying a facial reconstruction model, to the representation of the sending user depicting the sending user with an XR headset, the applying including:
encoding the representation of the sending user;
generating a predicted geometry by applying a geometry prediction branch of the facial reconstruction model to the encoded version of the representation of the sending user;
generating a predicted texture by applying a texture prediction branch of the facial reconstruction model to the encoded version of the representation of the sending user; and
skinning the predicted texture onto the predicted geometry,
wherein the facial reconstruction model is trained on training items, each training item generated by pairing an image of a user that does not depict the XR headset with a version of that image of the user with the XR headset superimposed and wherein the training includes:
providing the version of the image of the user, with the XR headset superimposed, to the facial reconstruction model, wherein the facial reconstruction model produces a predicted geometry skinned with a predicted texture; and
updating features of the facial reconstruction model based on a comparison of A) content based on the predicted geometry and the predicted texture with B) content based on the image of the user that does not depict the XR headset.
7. The non-transitory computer-readable storage medium of claim 6 , wherein training of the facial reconstruction model includes applying each particular training item, of the training items, to the facial reconstruction model:
wherein the providing the version of the image of the user to the facial reconstruction model includes encoding the version of the image of the user with the XR headset superimposed, from the particular training item;
wherein the facial reconstruction model generates the predicted geometry by applying a geometry prediction branch of the facial reconstruction model to the encoded version of the image;
wherein the updating features of the facial reconstruction model includes updating parameters of the geometry prediction branch, wherein the comparison includes a comparison of the predicted geometry and a geometry generated for the image of the user without the XR headset;
wherein the facial reconstruction model generates the predicted texture by applying a texture prediction branch of the facial reconstruction model to the encoded version of the image; and
wherein the updating features of the facial reconstruction model further includes updating parameters of the texture prediction branch, wherein the comparison further includes a comparison of the predicted texture and a texture generated for the image of the user without the XR headset.
8. The non-transitory computer-readable storage medium of claim 6 ,
wherein the facial reconstruction model includes a geometry branch and a texture branch; and
wherein the updating features of the facial reconstruction model includes updating parameters of the geometry prediction branch by applying multiple loss functions for L2 regularization, weighted parameter distance cost, and 3D vertex cost.
9. The non-transitory computer-readable storage medium of claim 6 ,
wherein the facial reconstruction model includes a geometry branch and a texture branch; and
wherein the updating features of the facial reconstruction model includes updating parameters of the texture prediction branch by applying multiple loss functions for L1 loss, SSIM loss, and/or perceptual loss.
10. The non-transitory computer-readable storage medium of claim 6 , wherein the encoding is performed by a convolutional neural network.
11. The non-transitory computer-readable storage medium of claim 6 , wherein the facial reconstruction model includes a geometry branch and a texture branch, and wherein the texture branch includes a decoder, that is a neural network with deconvolution layers having skip connections, that modifies an encoded input image to shape it for fitting onto a facial geometry.
12. The non-transitory computer-readable storage medium of claim 6 , wherein the facial reconstruction model includes a geometry branch and a texture branch, and wherein the geometry prediction branch includes a neural network that receives an encoded image and produces a predicted geometry of the face depicted in the encoded image.
13. The non-transitory computer-readable storage medium of claim 6 , wherein the facial reconstruction model is a recumbent machine learning model, using output of the facial reconstruction model from one or more previous iterations of facial reconstruction as input in a current iteration of facial reconstruction.
14. A computing system for performing a facial reconstruction on a representation of a sending user of a holograph call, the computing system comprising:
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising:
producing a version of the representation of the sending user without an XR headset or with the XR headset having transparent lenses by applying a facial reconstruction model, to the representation of the sending user depicting the sending user with an XR headset with non-transparent lenses, the applying including:
encoding the representation of the sending user;
generating a predicted geometry by applying a geometry prediction branch of the facial reconstruction model to the encoded version of the representation of the sending user;
generating a predicted texture by applying a texture prediction branch of the facial reconstruction model to the encoded version of the representation of the sending user; and
skinning the predicted texture onto the predicted geometry,
wherein the facial reconstruction model is trained on training items, each training item generated by pairing an image of a user that does not depict the XR headset or that depicts the XR headset with transparent lenses with a version of that image of the user with the XR headset superimposed without transparent lenses, wherein the training includes:
providing the version of the image of the user with the XR headset superimposed without transparent lenses, to the facial reconstruction model, wherein the facial reconstruction model produces a predicted geometry skinned with a predicted texture; and
updating features of the facial reconstruction model based on a comparison of A) content based on the predicted geometry and the predicted texture with B) content based on the image of the user that does not depict the XR headset or that depicts the XR headset with transparent lenses.
15. The computing system of claim 14 , wherein training of the facial reconstruction model includes applying each particular training item, of the training items, to the facial reconstruction model:
wherein the providing the version of the image of the user with the XR headset superimposed without transparent lenses, to the facial reconstruction model, includes encoding the version of the image of the user with the XR headset superimposed without transparent lenses, from the particular training item;
wherein the facial reconstruction model generates the predicted geometry by applying a geometry prediction branch of the facial reconstruction model to the encoded version of the image;
wherein the updating features of the facial reconstruction model includes updating parameters of the geometry prediction branch, wherein the comparison includes a comparison of the predicted geometry and a geometry generated for the image of the user that does not depict the XR headset or that depicts the XR headset with transparent lenses;
wherein the facial reconstruction model generates the predicted texture by applying a texture prediction branch of the facial reconstruction model to the encoded version of the image; and
wherein the updating features of the facial reconstruction model further includes updating parameters of the texture prediction branch, wherein the comparison further includes a comparison of the predicted texture and a texture generated for the image of the user that does not depict the XR headset or that depicts the XR headset with transparent lenses.
16. The computing system of claim 14 ,
wherein the facial reconstruction model includes a geometry branch and a texture branch;
wherein the updating features of the facial reconstruction model includes:
updating parameters of the geometry prediction branch by applying multiple loss functions for L2 regularization, weighted parameter distance cost, and/or 3D vertex cost; and
updating parameters of the texture prediction branch by applying multiple loss functions for L1 loss, SSIM loss, and/or perceptual loss.
17. The computing system of claim 14 , wherein the encoding is performed by a convolutional neural network.
18. The computing system of claim 14 ,
wherein the facial reconstruction model includes a geometry branch and a texture branch; and
wherein the texture branch includes a decoder, that is a neural network with deconvolution layers having skip connections, that warps an encoded input image to shape it for fitting onto a facial geometry.
19. The computing system of claim 14 ,
wherein the facial reconstruction model includes a geometry branch and a texture branch; and
wherein the geometry prediction branch includes a neural network that receives an encoded image and produces a predicted geometry of the face depicted in the encoded image.
20. The computing system of claim 14 , wherein the facial reconstruction model is a recumbent machine learning model, using output of the facial reconstruction model from one or more previous iterations of facial reconstruction as input in a current iteration of facial reconstruction.