Generating non-destructive synthetic lens blur with in-focus edge rendering
Methods, systems, and non-transitory computer readable storage media are disclosed for generating a lens blur effect in a digital image with in-focus edge rendering. The disclosed system generates a focal matte indicating an in-focus range of depth values of a digital image based on a focus region and a depth map of the digital image. The disclosed system generates a layered depth map comprising foreground depth values and background depth values of pixels across the digital image according to the depth map and the focal matte. The disclosed system also renders the digital image to include a lens blur effect by utilizing the focal matte and the layered depth map to determine a combination of the foreground depth values and the background depth values in connection with a splatting operation.
1 . A computer-implemented method comprising:
generating, by at least one processor, a focal matte indicating an in-focus range of depth values of a digital image based on a focus region and a depth map of the digital image;
generating, by the at least one processor, a layered depth map comprising foreground depth values and background depth values of pixels across the digital image according to the depth map and the focal matte; and
rendering, by the at least one processor, the digital image to include a lens blur effect by utilizing the focal matte and the layered depth map to determine a combination of the foreground depth values and the background depth values in connection with a splatting operation.
2 . The computer-implemented method of claim 1 , wherein generating the focal matte comprises:
providing, within a graphical user interface, a focal range selection tool comprising a sliding window with an adjustable range of depth values corresponding to a focus relative to a depth map of the digital image; and
determining the focus region in response to a position of the sliding window of the focal range selection tool.
3 . The computer-implemented method of claim 2 , wherein generating the focal matte comprises:
providing, within the graphical user interface, the focal range selection tool with one or more options to adjust a size of the sliding window; and
determining the focus region based further on the size of the sliding window.
4 . The computer-implemented method of claim 1 , wherein generating the focal matte comprises generating, for the digital image, a focus mask by thresholding depth values in the depth map according to the focus region.
5 . The computer-implemented method of claim 4 , wherein generating the focal matte comprises:
generating, for the digital image and based on the focus region, a trimap segmentation comprising one or more regions corresponding to discontinuities in the depth map; and
generating the focal matte by:
utilizing a matting model to generate focal matte values for an unknown region of the trimap segmentation; and
blending the focal matte values of the unknown region with the focus mask.
6 . The computer-implemented method of claim 1 , wherein generating the layered depth map comprises generating, utilizing the focal matte as a guide mask for a refinement machine-learning model, the foreground depth values and the background depth values over all pixels of the digital image according to the in-focus range of depth values.
7 . The computer-implemented method of claim 1 , wherein rendering the digital image to include the lens blur effect comprises:
generating a fused depth map comprising fused depth values of pixels in the digital image by utilizing the focal matte to generate a fused depth value by linearly interpolating between a foreground depth value and a background depth value corresponding to a particular pixel from the layered depth map; and
rendering the digital image utilizing the fused depth value of the particular pixel in connection with the splatting operation.
8 . The computer-implemented method of claim 7 , wherein rendering the digital image to include the lens blur effect comprises:
determining splat sizes of the splatting operation for the pixels of the digital image based on fused depth values of the fused depth map; and
generating the lens blur effect by integrating splat values for the pixels of the digital image according to the splat sizes.
9 . The computer-implemented method of claim 7 , wherein rendering the digital image to include the lens blur effect comprises:
determining, for the particular pixel, that a focal matte value from the focal matte is greater than a threshold value above zero; and
modulating a splat weight for the particular pixel by the focal matte value.
10 . A system comprising:
a memory component; and
one or more processing devices coupled to the memory component, the one or more processing devices to perform operations comprising:
providing, for display via a graphical user interface, a focal range selection tool comprising an adjustable range of depth values corresponding to a focus relative to a depth map of a digital image;
generating a focal matte indicating an in-focus range of depth values of a digital image based on a selected focus region via the focal range selection tool and a trimap segmentation corresponding to discontinuities in the depth map;
generating a layered depth map comprising a set of foreground depth values of pixels of the digital image and a set of background depth values of the pixels of the digital image according to the depth map and the focal matte; and
render the digital image with a lens blur effect by utilizing the focal matte to determine fused depth values by interpolating between the set of foreground depth values and the set of background depth values of the layered depth map in connection with a splatting operation.
11 . The system of claim 10 , wherein generating the focal matte comprises determining, in response to a user input via the focal range selection tool, a position of a sliding window indicating the adjustable range of depth values corresponding to the focus relative to the depth map.
12 . The system of claim 11 , wherein generating the focal matte comprises modifying a size of the adjustable range of depth values in response to an additional user input changing a position of an edge of the sliding window.
13 . The system of claim 10 , wherein generating the focal matte comprises:
generating a focus mask by thresholding depth values in the depth map according to the selected focus region;
generating, based on the focus mask and the depth map, the trimap segmentation indicating an unknown region corresponding to discontinuities in the focus mask; and
generating the focal matte by blending focal matte values of the unknown region of the trimap segmentation with the focus mask.
14 . The system of claim 10 , wherein generating the layered depth map comprises generating, utilizing a refinement machine-learning model guided by the focal matte, the set of foreground depth values in a foreground map and the set of background depth values in a background map, the foreground map overlapping the background map.
15 . The system of claim 10 , wherein rendering the digital image to include the lens blur effect comprises:
determining, for a pixel of the digital image, a splat size of the splatting operation based on a fused depth value corresponding to the pixel; and
generating the lens blur effect by integrating a splat value for the pixel determined according to the splat size with splat values of adjacent pixels.
16 . The system of claim 10 , wherein rendering the digital image to include the lens blur effect comprises:
determining, for a pixel of the digital image, that a focal matte value from the focal matte is greater than a threshold value; and
modulating a splat weight for the pixel by the focal matte value in connection with the splatting operation.
17 . A non-transitory computer readable medium storing executable instructions which, when executed by a processing device, cause the processing device to perform operations comprising:
determining, in response to an input via a graphical user interface, a focus mask comprising an in-focus range of depth values according to a depth map of a digital image indicated by a focus region;
generating a focal matte by refining edges of the focus mask according to discontinuities in the depth map;
generating a layered depth map comprising foreground depth values and background depth values of pixels of the digital image according to the depth map and the focal matte; and
rendering the digital image to include a lens blur effect by utilizing the focal matte and the layered depth map to determine a combination of the foreground depth values and the background depth values in connection with a splatting operation.
18 . The non-transitory computer readable medium of claim 17 , wherein the operations further comprise:
providing, within the graphical user interface, a focal range selection tool comprising a sliding window with an adjustable position and an adjustable size for indicating an adjustable range of depth values corresponding to a focus relative to a depth map of the digital image; and
determining the focus region based on a position and a size indicated by the sliding window of the focal range selection tool.
19 . The non-transitory computer readable medium of claim 17 , wherein generating the focal matte comprises:
generating, for the digital image, the focus mask by thresholding depth values in the depth map according to the focus region;
generating, for the digital image and based on the focus mask and the depth map, a trimap segmentation comprising an unknown region representing discontinuities in the depth map; and
generating the focal matte by blending focal matte values in the unknown region with the focus mask.
20 . The non-transitory computer readable medium of claim 17 , wherein rendering the digital image comprises:
determining, for the pixels of the digital image, splat sizes based on the combination of the foreground depth values and the background depth values according to the focal matte;
modulating one or more splat weights for one or more pixels according to corresponding focal matte values from the focal matte; and
generating the lens blur effect by generating splat values for the pixels of the digital image according to the splat sizes of the pixels of the digital image.