3d conversations in an artificial reality environment
A 3D conversation system can facilitate 3D conversations in an augmented reality environment, allowing conversation participants to appear as if they are face-to-face. The 3D conversation system can accomplish this with a pipeline of data processing stages, which can include calibrate, capture, tag and filter, compress, decompress, reconstruct, render, and display stages. Generally, the pipeline can capture images of the sending user, create intermediate representations, transform the representations to convert from the orientation the images were taken from to a viewpoint of the receiving user, and output images of the sending user, from the viewpoint of the receiving user, in synchronization with audio captured from the sending user. Such a 3D conversation can take place between two or more sender/receiving systems and, in some implementations can be mediated by one or more server systems. In various configurations, stages of the pipeline can be customized based on a conversation context.
1. A method for providing a 3D conversation, the method comprising:
obtaining 3D conversation data, at least partially from the one or more image capture devices, the 3D conversation data comprising color information and depth information;
generating a 3D representation based on the depth information;
compressing the color information and a first version of the 3D representation into a first version of transmission data; and
sending the first version of the transmission data to be delivered to a reconstruction system that:
receives the first version of the transmission data;
decompresses the first version of the transmission data into a second version of the transmission data including a second version of the 3D representation and color data; and
renders one or more 2D images from the second version of the 3D representation, wherein the one or more 2D images are further based on the color data; and
wherein the one or more 2D images are output by a recipient system engaged in the 3D conversation.
2. The method of claim 1 , wherein the output of the one or more 2D images includes a wearable projection system projecting light, based on the one or more 2D images, into at least one eye of a user of the recipient system.
3. The method of claim 1 , wherein the one or more image capture devices are selected from among multiple capture devices based on one or more of:
a determined relationship between a viewpoint of the selected one or more image capture devices and one or more viewpoints determined for the recipient system;
a determined compute capability of a sending system and/or of the recipient system;
a determined available amount of bandwidth;
a determined battery level;
determined display capabilities of the recipient system; or
any combination thereof.
4. The method of claim 1 further comprising assigning capture device identifiers to parts of the 3D conversation data according to the device that captured that part of the 3D conversation data;
wherein calibration data is associated with a camera identifier for which the calibration data was generated; and
wherein the method further comprises correlating the calibration data with the at least part of the 3D conversation data based on a match between A) the capture device identifier assigned to the at least part of the 3D conversation data and B) the camera identifier.
5. The method of claim 1 , wherein the compressing of at least part of the 3D conversation data comprises determining a type of the at least part of the 3D conversation data and selecting a compression algorithm specific to the determined type of the at least part of the 3D conversation data.
6. The method of claim 1 further comprising filtering portions of the 3D conversation data by:
distinguishing between a background area of the portions of the 3D conversation data and a depiction of a sending user; and
based on the distinguishing, removing from the 3D conversation data one or more depictions of the background area.
7. The method of claim 1 , wherein the depth data is generated based on captured image data.
8. The method of claim 1 , wherein the reconstruction system and the recipient system are components of the same computing device.
9. The method of claim 1 , wherein the reconstruction system is incorporated in a first computing device which sends the one or more 2D images to the recipient system, incorporated in a second computing device different from the first computing device, for the output.
10. The method of claim 1 , wherein the first version of transmission data was sent by a mobile phone device and wherein the recipient system is an artificial reality device comprising at least a wearable projection display that performs the outputting of the 2D images.
11. The method of claim 1 further comprising:
identifying conversation context factors comprising one or more of:
a determined compute capability of the recipient system;
a determined battery level;
determined display capabilities of the recipient system; or
any combination thereof; and
matching one or more factors of the conversation context to a 3D representation type in a hierarchy of 3D representation types;
wherein the generating the 3D representation comprises generating a 3D representation of the matched 3D representation type.
12. The method of claim 1 ,
wherein the first version of transmission data is associated with calibration data that specifies intrinsic parameters and extrinsic parameters of the one or more image capture devices that captured the 3D conversation data; and
wherein the generating the 3D representation comprises applying one or more transformations to the depth data based on the intrinsic parameters and extrinsic parameters to generate one or more of a point cloud, a signed distance function, populated voxels, a mesh, a light field; or any combination thereof.
13. A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for providing a 3D conversation, the process comprising:
obtaining 3D conversation data comprising color information and depth information;
generating a 3D representation based on the depth information;
compressing the color information and a first version of the 3D representation into a first version of transmission data; and
sending the first version of the transmission data to be delivered to a reconstruction system that:
receives the first version of the transmission data;
decompresses the first version of the transmission data into a second version of the transmission data including a second version of the 3D representation and color data; and
renders one or more 2D images from the second version of the 3D representation, wherein the one or more 2D images are further based on the color data; and
wherein the one or more 2D images are output by a recipient system engaged in the 3D conversation.
14. The computer-readable storage medium of claim 13 , wherein the process further comprises assigning capture device identifiers to parts of the 3D conversation data according to the device that captured that part of the 3D conversation data;
wherein calibration data is associated with a camera identifier for which the calibration data was generated; and
wherein the process further comprises correlating the calibration data with the at least part of the 3D conversation data based on a match between A) the capture device identifier assigned to the at least part of the 3D conversation data and B) the camera identifier.
15. The computer-readable storage medium of claim 13 , wherein the process further comprises filtering portions of the 3D conversation data by:
distinguishing between a background area of the portions of the 3D conversation data and a depiction of a sending user; and
based on the distinguishing, removing from the 3D conversation data one or more depictions of the background area.
16. The computer-readable storage medium of claim 13 , wherein the reconstruction system and the recipient system are components of the same computing device.
17. The computer-readable storage medium of claim 13 , wherein the first version of transmission data was sent by a mobile phone device and wherein the recipient system is an artificial reality device comprising at least a wearable projection display that performs the outputting of the 2D images.
18. A computing system for providing a 3D conversation, the computing system comprising:
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising:
obtaining 3D conversation data, at least partially from the one or more capture devices, the 3D conversation data comprising color information and depth information;
generating a 3D representation based on the depth information;
rendering, from the 3D representation and based on the color information, one or more 2D images;
compressing the one or more 2D images; and
sending the compressed one or more 2D images to be delivered to a recipient system that:
receives the compressed one or more 2D images;
decompresses the one or more 2D images; and
outputs the one or more 2D images.
19. The computing system of claim 18 , wherein the computing system is a mobile phone device and wherein the recipient system is an artificial reality device comprising at least a wearable projection display that performs the outputting of the 2D images.
20. The computing system of claim 18 , wherein the rendering the one or more 2D images from the 3D representation comprises determining capture angle, for the one or more 2D images taken from the 3D representation, based on a determined viewpoint for the recipient system.