Providing 3D data for messages in a messaging system
The subject technology receives, at a client device, a selection of a selectable graphical item from a plurality of selectable graphical items, the selectable graphical item comprising an augmented reality content generator including a 3D effect. The subject technology captures image data using at least one camera of the client device. The subject technology generates depth data using a machine learning model based at least in part on the captured image data. The subject technology applies, to the image data and the depth data, the 3D effect based at least in part on the augmented reality content generator.
1. A method, comprising:
receiving, at a client device, a selection of a selectable graphical item from a plurality of selectable graphical items, the selectable graphical item comprising an augmented reality content generator including a 3D effect;
capturing image data using at least one camera of the client device;
generating depth data using a machine learning model based at least in part on the captured image data; and
applying, to the image data and the depth data, the 3D effect based at least in part on the augmented reality content generator, the applying the 3D effect comprising:
generating a depth map using at least the depth data,
generating a packed depth map based at least in part on the depth map, the generating the packed depth map comprising:
converting a single channel floating point texture to a raw depth map, the raw depth map having a lower resolution than the image data, and
generating multiple channels based at least in part on the raw depth map,
generating a segmentation mask based at least on the image data, and
performing background inpainting and blurring of the image data using at least the segmentation mask to generate background inpainted image data, the performing the background inpainting comprising performing a diffusion based inpainting technique that fills in a missing region by propagating image content from a boundary between the missing region and a background region to an interior of the missing region, wherein the background region comprises a particular region of the image data without a foreground subject and the missing region includes the foreground subject.
2. The method of claim 1 , wherein the image data is captured with more than one camera or the image data is captured using dual pixel autofocus from a single camera.
3. The method of claim 1 , wherein the machine learning model comprises a deep neural network or a convolutional neural network that provides a prediction of depth data based on the captured image data.
4. The method of claim 3 , wherein the machine learning model receives the captured image data as an input, and generates a depth map as an output.
5. The method of claim 3 , wherein the machine learning model executes on a neural network processor or a graphics processing unit of the client device.
6. The method of claim 1 , further comprising:
generating a 3D message based at least in part on the applied 3D effect;
rendering a view of the 3D message based at least in part on the applied 3D effect;
receiving movement data from a movement sensor of the client device;
updating the view of the 3D message based at least in part on the received movement data; and
rendering the updated view of the 3D message.
7. The method of claim 1 , wherein the at least one camera comprises a first camera and a second camera, the first camera having a first focal length and the second camera having a second focal length, the first focal length and the second focal length being different.
8. The method of claim 1 , further comprising:
selecting a set of augmented reality content generators from a plurality of available augmented reality content generator based on metadata associated with each respective augmented reality content generator, the metadata including information indicating a corresponding augmented reality content generator includes at least a 3D effect, the set of augmented reality content generators including at least one augmented reality content generator without a 3D effect and at least one augmented reality content generator with a 3D effect.
9. The method of claim 8 , further comprising:
causing display of an interface comprising a plurality of selectable graphical items, each selectable graphical item corresponding to a respective augmented reality content generator of the set of augmented reality content generators.
10. A system comprising:
a processor; and
a memory including instructions that, when executed by the processor, cause the processor to perform operations comprising:
receiving, at a client device, a selection of a selectable graphical item from a plurality of selectable graphical items, the selectable graphical item comprising an augmented reality content generator including a 3D effect;
capturing image data using at least one camera of the client device;
generating depth data using a machine learning model based at least in part on the captured image data; and
applying, to the image data and the depth data, the 3D effect based at least in part on the augmented reality content generator, the applying the 3D effect comprising:
generating a depth map using at least the depth data,
generating a packed depth map based at least in part on the depth map, the generating the packed depth map comprising:
converting a single channel floating point texture to a raw depth map, the raw depth map having a lower resolution than the image data, and
generating multiple channels based at least in part on the raw depth map,
generating a segmentation mask based at least on the image data, and
performing background inpainting and blurring of the image data using at least the segmentation mask to generate background inpainted image data, the performing the background inpainting comprising performing a diffusion based inpainting technique that fills in a missing region by propagating image content from a boundary between the missing region and a background region to an interior of the missing region, wherein the background region comprises a particular region of the image data without a foreground subject and the missing region includes the foreground subject.
11. The system of claim 10 , wherein the image data is captured with more than one camera or the image data is captured using dual pixel autofocus from a single camera.
12. The system of claim 10 , wherein the machine learning model comprises a deep neural network or a convolutional neural network that provides a prediction of depth data based on the captured image data.
13. The system of claim 12 , wherein the machine learning model receives the captured image data as an input, and generates a depth map as an output.
14. The system of claim 12 , wherein the machine learning model executes on a neural network processor or a graphics processing unit of the client device.
15. The system of claim 10 , wherein the operations further comprise:
generating a 3D message based at least in part on the applied 3D effect;
rendering a view of the 3D message based at least in part on the applied 3D effect;
receiving movement data from a movement sensor of the client device;
updating the view of the 3D message based at least in part on the received movement data; and
rendering the updated view of the 3D message.
16. The system of claim 10 , wherein the at least one camera comprises a first camera and a second camera, the first camera having a first focal length and the second camera having a second focal length, the first focal length and the second focal length being different.
17. The system of claim 10 , wherein the operations further comprise:
selecting a set of augmented reality content generators from a plurality of available augmented reality content generator based on metadata associated with each respective augmented reality content generator, the metadata including information indicating a corresponding augmented reality content generator includes at least a 3D effect, the set of augmented reality content generators including at least one augmented reality content generator without a 3D effect and at least one augmented reality content generator with a 3D effect.
18. A non-transitory computer-readable medium comprising instructions, which when executed by a computing device, cause the computing device to perform operations comprising:
receiving, at a client device, a selection of a selectable graphical item from a plurality of selectable graphical items, the selectable graphical item comprising an augmented reality content generator including a 3D effect;
capturing image data using at least one camera of the client device;
generating depth data using a machine learning model based at least in part on the captured image data; and
applying, to the image data and the depth data, the 3D effect based at least in part on the augmented reality content generator, the applying the 3D effect comprising:
generating a depth map using at least the depth data,
generating a packed depth map based at least in part on the depth map, the generating the packed depth map comprising:
converting a single channel floating point texture to a raw depth map, the raw depth map having a lower resolution than the image data, and
generating multiple channels based at least in part on the raw depth map,
generating a segmentation mask based at least on the image data, and
performing background inpainting and blurring of the image data using at least the segmentation mask to generate background inpainted image data, the performing the background inpainting comprising performing a diffusion based inpainting technique that fills in a missing region by propagating image content from a boundary between the missing region and a background region to an interior of the missing region, wherein the background region comprises a particular region of the image data without a foreground subject and the missing region includes the foreground subject.