Generating 3D data in a messaging system
The subject technology receives image data and depth data. The subject technology selects an augmented reality content generator corresponding to a three-dimensional (3D) effect. The subject technology applies the 3D effect to the image data and the depth data based at least in part on the selected augmented reality content generator. The subject technology generates, using a processor, a message including information related to the applied 3D effect, the image data, and the depth data.
1. A method, comprising:
receiving, using a processor, image data and depth data;
selecting, using the processor, an augmented reality content generator corresponding to a three-dimensional (3D) effect;
applying, using the processor, the 3D effect to the image data and the depth data based at least in part on the selected augmented reality content generator, the applying the 3D effect comprising:
generating a depth map using at least the received depth data,
generating a segmentation mask based at least on the received image data, and
performing background inpainting and blurring of the received image data using at least the segmentation mask to generate background inpainted image data;
generating a packed depth map based at least in part on the depth map, the generating the packed depth map comprising:
converting a single channel floating point texture to a raw depth map, and
generating multiple channels based at least in part on the raw depth map; and
generating, using a processor, a message including information related to the applied 3D effect, the image data, and the depth data.
2. The method of claim 1 , wherein the multiple channels are lower precision than the single channel floating point texture and undergo multiple image processing operation without losing additional precision.
3. The method of claim 1 , further comprising:
generating a depth inpainting mask based at least in part on the depth map; and
performing inpainting of the depth map using the depth inpainting mask to generate an inpainted depth map.
4. The method of claim 3 , wherein the inpainted depth map extends a foreground region of the depth map, the extended foreground region being used to perform at least one image processing operation on a region of the image data corresponding to a representation of a portion of a face corresponding to hair, an ear, or a shoulder.
5. The method of claim 1 , further comprising:
generating a depth normal map based at least in part on the depth map; and
providing a post-process foreground image by applying, using the depth normal map, the 3D effect to a foreground region of the image data.
6. The method of claim 1 , further comprising:
sending the message to a messaging server for storing the message, the message including a set of assets, the set of assets comprising a post-processed foreground image, a post-processed depth map, and a background inpainted image.
7. The method of claim 6 , further comprising:
receiving a command to initiate sharing of the stored message to at least one receiving device; and
sending the stored message at least one receiving device.
8. A system comprising:
a processor; and
a memory including instructions that, when executed by the processor, cause the processor to perform operations comprising:
receiving, using a processor, image data and depth data;
selecting, using the processor, an augmented reality content generator corresponding to a three-dimensional (3D) effect;
applying, using the processor, the 3D effect to the image data and the depth data based at least in part on the selected augmented reality content generator, the applying the 3D effect comprising:
generating a depth map using at least the received depth data,
generating a segmentation mask based at least on the received image data, and
performing background inpainting and blurring of the received image data using at least the segmentation mask to generate background inpainted image data;
generating a packed depth map based at least in part on the depth map, the generating the packed depth map comprising:
converting a single channel floating point texture to a raw depth map, and
generating multiple channels based at least in part on the raw depth map; and
generating, using a processor, a message including information related to the applied 3D effect, the image data, and the depth data.
9. The system of claim 8 , wherein the multiple channels are lower precision than the single channel floating point texture and undergo multiple image processing operation without losing additional precision.
10. The system of claim 8 , wherein the operations further comprise:
generating a depth inpainting mask based at least in part on the depth map; and
performing inpainting of the depth map using the depth inpainting mask to generate an inpainted depth map.
11. The system of claim 10 , wherein the inpainted depth map extends a foreground region of the depth map, the extended foreground region being used to perform at least one image processing operation on a region of the image data corresponding to a representation of a portion of a face corresponding to hair, an ear, or a shoulder.
12. The system of claim 8 , wherein the operations further comprise:
generating a depth normal map based at least in part on the depth map; and
providing a post-process foreground image by applying, using the depth normal map, the 3D effect to a foreground region of the image data.
13. The system of claim 8 , wherein the operations further comprise:
sending the message to a messaging server for storing the message, the message including a set of assets, the set of assets comprising a post-processed foreground image, a post-processed depth map, and a background inpainted image.
14. A non-transitory computer-readable medium comprising instructions, which when executed by a computing device, cause the computing device to perform operations comprising:
receiving, using a processor, image data and depth data;
selecting, using the processor, an augmented reality content generator corresponding to a three-dimensional (3D) effect;
applying, using the processor, the 3D effect to the image data and the depth data based at least in part on the selected augmented reality content generator, the applying the 3D effect comprising:
generating a depth map using at least the received depth data,
generating a segmentation mask based at least on the received image data, and
performing background inpainting and blurring of the received image data using at least the segmentation mask to generate background inpainted image data;
generating a packed depth map based at least in part on the depth map, the generating the packed depth map comprising:
converting a single channel floating point texture to a raw depth map, and
generating multiple channels based at least in part on the raw depth map; and
generating, using a processor, a message including information related to the applied 3D effect, the image data, and the depth data.