Effects for 3D data in a messaging system

1. A method, comprising:

selecting a set of augmented reality content generators from a plurality of available augmented reality content generator based on metadata associated with each respective augmented reality content generator, the metadata including information indicating a corresponding augmented reality content generator includes at least a 3D effect, the set of augmented reality content generators including at least one augmented reality content generator without a 3D effect and at least one augmented reality content generator with a 3D effect;

receiving, at a client device, a selection of a selectable graphical item from a plurality of selectable graphical items, the selectable graphical item comprising an augmented reality content generator including a 3D effect;

capturing image data and depth data using at least one camera of the client device; and

applying, to the image data and the depth data, the 3D effect based at least in part on the augmented reality content generator, the applying the 3D effect comprising:

generating a depth map using at least the depth data,

generating a segmentation mask based at least on the image data, and

performing background inpainting and blurring of the image data using at least the segmentation mask to generate background inpainted image data, the performing the background inpainting comprising performing a diffusion based inpainting technique that fills in a missing region by propagating image content from a boundary between the missing region and a background region to an interior of the missing region, wherein the background region comprises a particular region of the image data without a foreground subject and the missing region includes the foreground subject, generating the depth map comprises converting a single channel floating point texture into a raw depth map, portions of the single channel floating point texture is sent into multiple lower precision channels, the raw depth map has a lower resolution than the image data;

generating a 3D message based at least in part on the applied 3D effect; and

rendering a view of the 3D message based at least in part on the applied 3D effect.

2. The method of claim 1 , wherein the at least one camera comprises a first camera and a second camera, the first camera having a first focal length and the second camera having a second focal length, the first focal length and the second focal length being different.

3. The method of claim 2 , wherein a disparity map is generated based at least in part on a distance between a first pixel from a first image captured by the first camera and a second pixel from a second image captured by the second camera, the first pixel and second pixel corresponding to a same object.

4. The method of claim 3 , wherein the disparity map comprises an image where each pixel includes a distance value between a pixel from the first image to corresponding pixel from the second image.

5. The method of claim 4 , wherein first pixels of a first object in the disparity map have a greater brightness than second pixels of a second object in the disparity map, the first pixels having a lesser depth values than second depth values of the second pixels.

6. The method of claim 3 , wherein the depth map is generated based at least in part on the disparity map.

7. The method of claim 1 , wherein the augmented reality content generator includes a beautification operation.

8. The method of claim 7 , wherein the augmented reality content generator includes a 3D object rendered in proximity to facial image data from the image data.

9. The method of claim 1 , further comprising:

causing display of an interface comprising a plurality of selectable graphical items, each selectable graphical corresponding to a respective augmented reality content generator of the set of augmented reality content generators.

10. A system comprising:

a processor; and

a memory including instructions that, when executed by the processor, cause the processor to perform operations comprising:

selecting a set of augmented reality content generators from a plurality of available augmented reality content generator based on metadata associated with each respective augmented reality content generator, the metadata including information indicating a corresponding augmented reality content generator includes at least a 3D effect, the set of augmented reality content generators including at least one augmented reality content generator without a 3D effect and at least one augmented reality content generator with a 3D effect;

receiving, at a client device, a selection of a selectable graphical item from a plurality of selectable graphical items, the selectable graphical item comprising an augmented reality content generator including a 3D effect;

capturing image data and depth data using at least one camera of the client device; and

applying, to the image data and the depth data, the 3D effect based at least in part on the augmented reality content generator, the applying the 3D effect comprising:

generating a depth map using at least the depth data,

generating a segmentation mask based at least on the image data, and

performing background inpainting and blurring of the image data using at least segmentation mask to generate background inpainted image data, the performing the background inpainting comprising performing a diffusion based inpainting technique that fills in a missing region by propagating image content from a boundary between the missing region and a background region to an interior of the missing region, wherein the background region comprises a particular region of the image data without a foreground subject and the missing region includes the foreground subject, generating the depth map comprises converting a single channel floating point texture into a raw depth map, portions of the single channel floating point texture is sent into multiple lower precision channels, the raw depth map has a lower resolution than the image data;

generating a 3D message based at least in part on the applied 3D effect; and

rendering a view of the 3D message based at least in part on the applied 3D effect.

11. The system of claim 10 , wherein the at least one camera comprises a first camera and a second camera, the first camera having a first focal length and the second camera having a second focal length, the first focal length and the second focal length being different.

12. The system of claim 11 , wherein a disparity map is generated based at least in part on a distance between a first pixel from a first image captured by the first camera and a second pixel from a second image captured by the second camera, the first pixel and second pixel corresponding to a same object.

13. The system of claim 12 , wherein the disparity map comprises an image where each pixel includes a distance value between a pixel from the first image to corresponding pixel from the second image.

14. The system of claim 13 , wherein first pixels of a first object in the disparity map have a greater brightness than second pixels of a second object in the disparity map, the first pixels having a lesser depth values than second depth values of the second pixels.

15. The system of claim 12 , wherein the depth map is generated based at least in part on the disparity map.

16. The system of claim 10 , wherein the augmented reality content generator includes a beautification operation.

17. The system of claim 10 , wherein the augmented reality content generator includes a 3D object rendered in proximity to facial image data from the image data.

18. A non-transitory computer-readable medium comprising instructions, which when executed by a computing device, cause the computing device to perform operations comprising:

selecting a set of augmented reality content generators from a plurality of available augmented reality content generator based on metadata associated with each respective augmented reality content generator, the metadata including information indicating a corresponding augmented reality content generator includes at least a 3D effect, the set of augmented reality content generators including at least one augmented reality content generator without a 3D effect and at least one augmented reality content generator with a 3D effect;

receiving, at a client device, a selection of a selectable graphical item from a plurality of selectable graphical items, the selectable graphical item comprising an augmented reality content generator including a 3D effect;

capturing image data and depth data using at least one camera of the client device; and

applying, to the image data and the depth data, the 3D effect based at least in part on the augmented reality content generator, the applying the 3D effect comprising:

generating a depth map using at least the depth data,

generating a segmentation mask based at least on the image data, and

performing background inpainting and blurring of the image data using at least the segmentation mask to generate background inpainted image data, the performing the background inpainting comprising performing a diffusion based inpainting technique that fills in a missing region by propagating image content from a boundary between the missing region and a background region to an interior of the missing region, wherein the background region comprises a particular region of the image data without a foreground subject and the missing region includes the foreground subject, generating the depth map comprises converting a single channel floating point texture into a raw depth map, portions of the single channel floating point texture is sent into multiple lower precision channels, the raw depth map has a lower resolution than the image data;

generating a 3D message based at least in part on the applied 3D effect; and

rendering a view of the 3D message based at least in part on the applied 3D effect.