Image composition for extended reality systems
Implementations of the subject technology provide for image composition for extended reality systems. Image composition may include combining virtual content from virtual images with physical content from images captured by one or more cameras. The virtual content and the physical content can be combined to form a composite image using depth information for the virtual content and the physical content. An adjustment mask may be generated to indicate edges or boundaries between virtual and physical content at which artifact correction for the composite image can be performed.
1. A method, comprising:
obtaining a virtual object and corresponding virtual depth information;
capturing an image of at least one physical object and corresponding physical depth information;
generating an occlusion mask based at least in part on the virtual depth information and the physical depth information;
generating an adjustment mask based at least in part on the occlusion mask and the virtual depth information; and
generating a composite image including at least a portion of the virtual object and at least a portion of the image using the occlusion mask and the adjustment mask.
2. The method of claim 1 , wherein the occlusion mask indicates pixel locations at which the physical object is foreground to the virtual object.
3. The method of claim 2 , further comprising generating an edge mask based on the virtual depth information, wherein the edge mask indicates locations at one or more edges of the virtual object.
4. The method of claim 3 , further comprising:
dilating the edge mask prior to generating the adjustment mask; and
generating the adjustment mask based on the occlusion mask and the dilated edge mask.
5. The method of claim 4 , wherein generating the adjustment mask based on the occlusion mask and the dilated edge mask comprises:
dilating the occlusion mask prior to generating the adjustment mask; and
generating the adjustment mask based on the dilated occlusion mask and the dilated edge mask.
6. The method of claim 1 , wherein generating the composite image including the at least the portion of the virtual object and the at least the portion of the image using the occlusion mask and the adjustment mask comprises:
generating an alpha mask based on the occlusion mask; and
modifying one or more parameters of the alpha mask based on the adjustment mask.
7. The method of claim 6 , wherein generating the composite image further comprises:
forming pixel values of the composite image from one or more of corresponding pixel values of a virtual image and corresponding pixel values of the image, as determined based on corresponding parameters for corresponding pixels of the alpha mask.
8. The method of claim 1 , further comprising:
splitting the adjustment mask to form an interior adjustment mask and an exterior adjustment mask,
wherein generating the composite image using the occlusion mask and the adjustment mask comprises generating the composite image using the occlusion mask and either the interior adjustment mask or the exterior adjustment mask.
9. The method of claim 8 , wherein splitting the adjustment mask comprises:
generating the interior adjustment mask based on the occlusion mask and the adjustment mask;
generating an inverse of the occlusion mask; and
generating the exterior adjustment mask based on the inverse of the occlusion mask and the adjustment mask.
10. The method of claim 1 , further comprising:
splitting the adjustment mask to form an interior adjustment mask and an exterior adjustment mask,
wherein generating the composite image using the occlusion mask and the adjustment mask comprises generating the composite image using the occlusion mask and both the interior adjustment mask and the exterior adjustment mask.
11. The method of claim 10 , wherein generating the composite image using the occlusion mask and both the interior adjustment mask and the exterior adjustment mask comprises:
performing a correction associated with the image using the interior adjustment mask; and
performing a correction associated with the virtual object using the exterior adjustment mask.
12. A device, comprising:
a camera;
memory; and
one or more processors configured to:
obtain a virtual object and corresponding virtual depth information;
capture, using at least the camera, an image of at least one physical object and corresponding physical depth information;
generate an occlusion mask based at least in part on the virtual depth information and the physical depth information;
generate an adjustment mask based at least in part on the occlusion mask and the virtual depth information; and
generate a composite image including at least a portion of the virtual object and at least a portion of the image using the occlusion mask and the adjustment mask.
13. The device of claim 12 , further comprising a depth sensor configured to obtain the physical depth information, and a display configured to display the composite image.
14. The device of claim 12 , wherein the one or more processors are configured to generate the composite image including the at least the portion of the virtual object and the at least the portion of the image using the occlusion mask and the adjustment mask by:
generating an alpha mask based on the occlusion mask; and
modifying one or more parameters of the alpha mask based on the adjustment mask.
15. The device of claim 14 , wherein the one or more processors are further configured to generate the composite image by forming pixel values of the composite image from one or more of corresponding pixel values of a virtual image and corresponding pixel values of the image, as determined based on corresponding parameters for corresponding pixels of the alpha mask.
16. The device of claim 12 , wherein the one or more processors are further configured to:
generate an edge mask based on the virtual depth information;
dilate the edge mask prior to generating the adjustment mask; and
generate the adjustment mask based on the occlusion mask and the dilated edge mask.
17. The device of claim 16 , wherein the one or more processors are configured to generate the adjustment mask based on the occlusion mask and the dilated edge mask by:
dilating the occlusion mask prior to generating the adjustment mask; and
generating the adjustment mask based on the dilated occlusion mask and the dilated edge mask.
18. A non-transitory computer-readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:
obtaining a virtual object and corresponding virtual depth information;
capturing an image of at least one physical object and corresponding physical depth information;
generating an occlusion mask based at least in part on the virtual depth information and the physical depth information;
generating an adjustment mask based at least in part on the occlusion mask and the virtual depth information; and
generating a composite image including at least a portion of the virtual object and at least a portion of the image using the occlusion mask and the adjustment mask.
19. The non-transitory computer-readable medium of claim 18 , wherein the operations further comprise:
generating an edge mask based on the virtual depth information;
dilating the edge mask prior to generating the adjustment mask; and
generating the adjustment mask based on the occlusion mask and the dilated edge mask.
20. The non-transitory computer-readable medium of claim 19 , wherein generating the adjustment mask based on the occlusion mask and the dilated edge mask includes:
dilating the occlusion mask prior to generating the adjustment mask; and
generating the adjustment mask based on the dilated occlusion mask and the dilated edge mask.
21. The non-transitory computer-readable medium of claim 18 , wherein generating the composite image including the at least the portion of the virtual object and the at least the portion of the image using the occlusion mask and the adjustment mask comprises:
generating an alpha mask based on the occlusion mask;
modifying one or more parameters of the alpha mask based on the adjustment mask; and
forming pixel values of the composite image from one or more of corresponding pixel values of a virtual image and corresponding pixel values of the image, as determined based on corresponding parameters for corresponding pixels of the alpha mask.
22. The non-transitory computer-readable medium of claim 18 , wherein the operations further comprise:
splitting the adjustment mask to form an interior adjustment mask and an exterior adjustment mask, wherein generating the composite image using the occlusion mask and the adjustment mask comprises generating the composite image using the occlusion mask and both the interior adjustment mask and the exterior adjustment mask.
23. The method of claim 1 , wherein the adjustment mask identifies locations in the composite image at which a correction is to be applied to correct an artifact generated by using the occlusion mask.
24. The method of claim 23 , wherein generating the composite image comprises adjusting one or more portions of the composite image to remove the artifact by modifying pixels of the composite image that are identified by the adjustment mask.
25. The method of claim 6 , wherein modifying the one or more parameters of the alpha mask based on the adjustment mask comprises modifying one or more weighting coefficients of one or more pixels of the alpha mask at locations in the alpha mask that correspond to non-zero pixels of the adjustment mask.
26. The method of claim 1 , wherein generating the composite image including the at least the portion of the virtual object and the at least the portion of the image using the occlusion mask and the adjustment mask comprises:
generating an alpha mask based on the occlusion mask, the alpha mask comprising alpha mask values for respective alpha mask pixels, each of the alpha mask pixels corresponding to a pixel of the composite image; and
generating at least one pixel value for at least one pixel of the composite image by overriding the alpha mask value of the alpha mask pixel that corresponds to the at least one pixel of the composite image.
27. The method of claim 26 , wherein overriding the alpha mask value of the alpha mask pixel that corresponds to the at least one pixel of the composite image comprises ignoring the alpha mask value of the alpha mask pixel that corresponds to the at least one pixel of the composite image and selecting a pixel value from the virtual object or a pixel value from the image as the at least one pixel value for the at least one pixel of the composite image.
28. The method of claim 5 , wherein generating the adjustment mask based on the dilated occlusion mask and the dilated edge mask comprises computing a logical AND of the dilated edge mask and the dilated occlusion mask.